2024-04-20T02:31:09.864 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-20T02:31:09.864 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-snaps} --verbose --name yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272 -- /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272/orig.config.yaml 2024-04-20T02:31:09.910 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-20T02:31:10.003 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272 branch: wip-yuri7-testing-2024-04-18-1351-reef description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-snaps} email: yweinste@redhat.com first_in_suite: false job_id: '7664272' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri7-testing-2024-04-18-1351-reef ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 2500 ms tcp read timeout: 5 ms type: async osd_async_recovery_min_cost: 1 osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_object_clean_region_max_num_intervals: 1000 osd_pg_log_trim_min: 0 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd max scrubs: 3 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - \(POOL_APP_NOT_ENABLED\) sha1: 15eecde2bab9803889d929c7e423d07970d8cbed ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 15eecde2bab9803889d929c7e423d07970d8cbed selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 min_in: 4 workunit: branch: wip-yuri7-testing-2024-04-18-1351-reef sha1: 15eecde2bab9803889d929c7e423d07970d8cbed owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6867 sha1: 15eecde2bab9803889d929c7e423d07970d8cbed sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri7-testing-2024-04-18-1351-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_15eecde2bab9803889d929c7e423d07970d8cbed/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 15eecde2bab9803889d929c7e423d07970d8cbed targets: smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL1jsavjLcnnMFDvHQQVo4wyxEtGjfg3I1vllmWLMvSUwzstsJSDc7LOUC9yO94vHnJq4GyODKZtuW4kVlSj9eY= smithi138.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAAMtHnh0m2QRjWnLyeuXiMSQwCODZQsnGwUbwKI81GUnu0f0TYE8o6NQJjkZG0RQWj/IBq/B8j+aAq1Y3zl+GQ= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 dedup_chunk_algo: fastcdc dedup_chunk_size: '131072' enable_dedup: true low_tier_pool: low_tier objects: 50 op_weights: read: 100 rollback: 10 set_chunk: 30 snap_create: 10 snap_remove: 10 tier_evict: 10 tier_flush: 5 tier_promote: 10 write: 50 ops: 1500 set_chunk: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-20_01:10:46 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-20T02:31:10.003 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_15eecde2bab9803889d929c7e423d07970d8cbed/qa; will attempt to use it 2024-04-20T02:31:10.004 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_15eecde2bab9803889d929c7e423d07970d8cbed/qa/tasks 2024-04-20T02:31:10.004 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-20T02:31:10.006 INFO:teuthology.task.internal:Checking packages... 2024-04-20T02:31:10.031 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '15eecde2bab9803889d929c7e423d07970d8cbed' 2024-04-20T02:31:10.031 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-20T02:31:10.031 INFO:teuthology.packaging:ref: None 2024-04-20T02:31:10.031 INFO:teuthology.packaging:tag: None 2024-04-20T02:31:10.031 INFO:teuthology.packaging:branch: wip-yuri7-testing-2024-04-18-1351-reef 2024-04-20T02:31:10.031 INFO:teuthology.packaging:sha1: 15eecde2bab9803889d929c7e423d07970d8cbed 2024-04-20T02:31:10.032 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri7-testing-2024-04-18-1351-reef 2024-04-20T02:31:10.177 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1252.g15eecde2 2024-04-20T02:31:10.179 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-20T02:31:10.187 INFO:teuthology.task.internal:no buildpackages task found 2024-04-20T02:31:10.187 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-20T02:31:10.240 INFO:teuthology.task.internal:Saving configuration 2024-04-20T02:31:10.256 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-20T02:31:10.262 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-20T02:31:10.286 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 02:26:10.972259', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL1jsavjLcnnMFDvHQQVo4wyxEtGjfg3I1vllmWLMvSUwzstsJSDc7LOUC9yO94vHnJq4GyODKZtuW4kVlSj9eY='} 2024-04-20T02:31:10.306 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-20 02:26:10.971234', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAAMtHnh0m2QRjWnLyeuXiMSQwCODZQsnGwUbwKI81GUnu0f0TYE8o6NQJjkZG0RQWj/IBq/B8j+aAq1Y3zl+GQ='} 2024-04-20T02:31:10.306 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-20T02:31:10.311 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-04-20T02:31:10.312 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-20T02:31:10.312 INFO:teuthology.run_tasks:Running task console_log... 2024-04-20T02:31:10.387 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fde803baf70>, signals=[15]) 2024-04-20T02:31:10.390 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-20T02:31:10.395 INFO:teuthology.task.internal:Opening connections... 2024-04-20T02:31:10.395 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:31:10.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T02:31:10.476 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:31:10.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T02:31:10.548 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-20T02:31:10.553 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-04-20T02:31:10.574 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-04-20T02:31:10.574 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-04-20T02:31:10.635 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="Red Hat Enterprise Linux" 2024-04-20T02:31:10.635 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="8.6 (Ootpa)" 2024-04-20T02:31:10.635 INFO:teuthology.orchestra.run.smithi105.stdout:ID="rhel" 2024-04-20T02:31:10.635 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="fedora" 2024-04-20T02:31:10.635 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="8.6" 2024-04-20T02:31:10.635 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el8" 2024-04-20T02:31:10.635 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-20T02:31:10.635 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-04-20T02:31:10.636 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-20T02:31:10.636 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://www.redhat.com/" 2024-04-20T02:31:10.636 INFO:teuthology.orchestra.run.smithi105.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-20T02:31:10.636 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-20T02:31:10.636 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:31:10.636 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-20T02:31:10.636 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-20T02:31:10.636 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-20T02:31:10.636 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-20T02:31:10.637 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-04-20T02:31:10.659 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2024-04-20T02:31:10.679 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2024-04-20T02:31:10.679 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-04-20T02:31:10.742 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="Red Hat Enterprise Linux" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8.6 (Ootpa)" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:ID="rhel" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="fedora" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8.6" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://www.redhat.com/" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-20T02:31:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:31:10.744 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-20T02:31:10.744 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-20T02:31:10.744 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-20T02:31:10.744 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-20T02:31:10.744 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-04-20T02:31:10.767 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-20T02:31:10.779 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-20T02:31:10.785 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-20T02:31:10.786 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-04-20T02:31:10.788 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2024-04-20T02:31:10.806 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-20T02:31:10.812 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-20T02:31:10.813 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-04-20T02:31:10.849 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2024-04-20T02:31:10.890 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-20T02:31:10.948 INFO:teuthology.run_tasks:Running task kernel... 2024-04-20T02:31:10.962 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-20T02:31:10.963 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-20T02:31:10.963 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-20T02:31:10.963 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-20T02:31:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:31:10.982 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-04-20T02:31:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:31:10.983 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-04-20T02:31:11.043 INFO:teuthology.orchestra.run.smithi105.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-20T02:31:11.043 INFO:teuthology.task.kernel:Running kernel on smithi105: 4.18.0-513.9.1.el8_9.x86_64 2024-04-20T02:31:11.043 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2024-04-20T02:31:11.045 INFO:teuthology.orchestra.run.smithi138.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-20T02:31:11.045 INFO:teuthology.task.kernel:Running kernel on smithi138: 4.18.0-513.9.1.el8_9.x86_64 2024-04-20T02:31:11.045 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum install -y kernel 2024-04-20T02:31:12.575 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:31:12.575 INFO:teuthology.orchestra.run.smithi138.stdout:Unable to read consumer identity 2024-04-20T02:31:12.578 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:31:12.579 INFO:teuthology.orchestra.run.smithi105.stdout:Unable to read consumer identity 2024-04-20T02:31:12.838 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:31:12.838 INFO:teuthology.orchestra.run.smithi105.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-20T02:31:12.838 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:31:12.873 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:31:12.874 INFO:teuthology.orchestra.run.smithi138.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-20T02:31:12.874 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:31:14.677 INFO:teuthology.orchestra.run.smithi105.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2024-04-20T02:31:14.761 INFO:teuthology.orchestra.run.smithi138.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-04-20T02:31:16.653 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 16 MB 00:01 2024-04-20T02:31:20.370 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 253 kB/s | 24 kB 00:00 2024-04-20T02:31:21.147 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-20T02:31:21.147 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-20T02:31:21.214 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:31:21.215 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-20T02:31:21.215 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:31:21.378 DEBUG:teuthology.orchestra.run.smithi105:> echo no | sudo yum reinstall kernel || true 2024-04-20T02:31:21.712 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:31:21.713 INFO:teuthology.orchestra.run.smithi105.stdout:Unable to read consumer identity 2024-04-20T02:31:21.916 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:31:21.916 INFO:teuthology.orchestra.run.smithi105.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-20T02:31:21.916 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:31:22.226 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:03 ago on Sat 20 Apr 2024 02:31:19 AM UTC. 2024-04-20T02:31:22.308 INFO:teuthology.orchestra.run.smithi105.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-20T02:31:22.309 INFO:teuthology.orchestra.run.smithi105.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-20T02:31:22.369 INFO:teuthology.orchestra.run.smithi105.stderr:Error: No packages marked for reinstall. 2024-04-20T02:31:22.416 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum reinstall -y kernel || true 2024-04-20T02:31:22.750 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:31:22.750 INFO:teuthology.orchestra.run.smithi105.stdout:Unable to read consumer identity 2024-04-20T02:31:22.962 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:31:22.962 INFO:teuthology.orchestra.run.smithi105.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-20T02:31:22.962 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:31:23.266 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:04 ago on Sat 20 Apr 2024 02:31:19 AM UTC. 2024-04-20T02:31:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-20T02:31:23.329 INFO:teuthology.orchestra.run.smithi105.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-20T02:31:23.359 INFO:teuthology.orchestra.run.smithi105.stderr:Error: No packages marked for reinstall. 2024-04-20T02:31:23.406 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T02:31:23.562 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-20T02:31:23.562 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-20T02:31:23.562 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-20T02:31:23.562 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-20T02:31:23.563 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-20T02:31:23.603 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2024-04-20T02:31:23.615 DEBUG:teuthology.parallel:result is None 2024-04-20T02:31:28.878 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 1.2 MB/s | 16 MB 00:14 2024-04-20T02:31:32.487 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 337 kB/s | 24 kB 00:00 2024-04-20T02:31:33.214 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-20T02:31:33.214 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-20T02:31:33.293 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:31:33.294 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-04-20T02:31:33.295 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:31:33.459 DEBUG:teuthology.orchestra.run.smithi138:> echo no | sudo yum reinstall kernel || true 2024-04-20T02:31:33.798 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:31:33.799 INFO:teuthology.orchestra.run.smithi138.stdout:Unable to read consumer identity 2024-04-20T02:31:34.031 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:31:34.031 INFO:teuthology.orchestra.run.smithi138.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-20T02:31:34.031 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:31:34.341 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:02 ago on Sat 20 Apr 2024 02:31:32 AM UTC. 2024-04-20T02:31:34.418 INFO:teuthology.orchestra.run.smithi138.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-20T02:31:34.418 INFO:teuthology.orchestra.run.smithi138.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-20T02:31:34.459 INFO:teuthology.orchestra.run.smithi138.stderr:Error: No packages marked for reinstall. 2024-04-20T02:31:34.507 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum reinstall -y kernel || true 2024-04-20T02:31:34.840 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:31:34.840 INFO:teuthology.orchestra.run.smithi138.stdout:Unable to read consumer identity 2024-04-20T02:31:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:31:35.126 INFO:teuthology.orchestra.run.smithi138.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-20T02:31:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:31:35.430 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:03 ago on Sat 20 Apr 2024 02:31:32 AM UTC. 2024-04-20T02:31:35.493 INFO:teuthology.orchestra.run.smithi138.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-20T02:31:35.493 INFO:teuthology.orchestra.run.smithi138.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-20T02:31:35.534 INFO:teuthology.orchestra.run.smithi138.stderr:Error: No packages marked for reinstall. 2024-04-20T02:31:35.586 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel | sort -rV | head -n 1 2024-04-20T02:31:35.778 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-20T02:31:35.778 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-20T02:31:35.778 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-20T02:31:35.779 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-20T02:31:35.779 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-20T02:31:35.812 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2024-04-20T02:31:35.828 DEBUG:teuthology.parallel:result is None 2024-04-20T02:31:35.828 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-20T02:31:35.835 INFO:teuthology.task.internal:Creating test directory... 2024-04-20T02:31:35.835 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-20T02:31:35.839 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-20T02:31:35.885 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-20T02:31:35.898 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-20T02:31:36.076 INFO:teuthology.task.internal:Creating archive directory... 2024-04-20T02:31:36.076 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-20T02:31:36.079 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-20T02:31:36.126 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-20T02:31:36.133 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-20T02:31:36.133 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-20T02:31:36.150 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-20T02:31:36.186 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T02:31:36.197 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T02:31:36.198 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T02:31:36.207 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-20T02:31:36.209 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-20T02:31:36.216 INFO:teuthology.task.internal:Configuring sudo... 2024-04-20T02:31:36.216 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-20T02:31:36.242 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-20T02:31:36.276 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-20T02:31:36.285 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-20T02:31:36.285 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-20T02:31:36.313 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-20T02:31:36.333 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T02:31:36.419 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T02:31:36.491 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T02:31:36.537 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T02:31:36.581 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:31:36.581 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-20T02:31:36.650 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T02:31:36.693 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-20T02:31:36.795 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T02:31:36.841 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-20T02:31:36.882 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:31:36.883 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-20T02:31:36.950 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-04-20T02:31:36.952 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2024-04-20T02:31:37.012 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T02:31:37.040 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T02:31:37.420 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-20T02:31:37.430 INFO:teuthology.task.internal:Starting timer... 2024-04-20T02:31:37.430 INFO:teuthology.run_tasks:Running task pcp... 2024-04-20T02:31:37.443 INFO:teuthology.run_tasks:Running task selinux... 2024-04-20T02:31:37.452 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-20T02:31:37.453 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-04-20T02:31:37.512 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-04-20T02:31:37.514 DEBUG:teuthology.orchestra.run.smithi138:> sudo service auditd rotate 2024-04-20T02:31:37.577 INFO:teuthology.orchestra.run.smithi138.stdout:Rotating logs: 2024-04-20T02:31:37.578 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-20T02:31:37.578 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-04-20T02:31:37.605 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-04-20T02:31:37.605 DEBUG:teuthology.orchestra.run.smithi138:> /usr/sbin/getenforce 2024-04-20T02:31:37.626 INFO:teuthology.orchestra.run.smithi138.stdout:Permissive 2024-04-20T02:31:37.627 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi105.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi138.front.sepia.ceph.com': 'permissive'} 2024-04-20T02:31:37.627 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-20T02:31:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:31:37.674 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-20T02:31:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:31:37.701 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-20T02:31:37.702 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-04-20T02:31:37.742 DEBUG:teuthology.orchestra.run.smithi138:> sudo /usr/sbin/setenforce permissive 2024-04-20T02:31:37.769 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-20T02:31:37.779 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-20T02:31:37.796 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-20T02:31:37.798 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi105.front.sepia.ceph.com,smithi138.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-20T02:41:39.448 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi105.front.sepia.ceph.com'), Remote(name='ubuntu@smithi138.front.sepia.ceph.com')] 2024-04-20T02:41:39.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-20T02:41:39.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T02:41:39.548 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-04-20T02:41:39.575 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-20T02:41:39.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-04-20T02:41:39.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-20T02:41:39.647 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-04-20T02:41:39.672 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-04-20T02:41:39.672 INFO:teuthology.run_tasks:Running task clock... 2024-04-20T02:41:39.683 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-20T02:41:39.683 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-20T02:41:39.683 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T02:41:39.685 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-20T02:41:39.685 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T02:41:39.715 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-20T02:41:39.729 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-20T02:41:39.743 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-20T02:41:39.758 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-20T02:41:39.802 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-04-20T02:41:39.814 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-04-20T02:41:39.829 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-20T02:41:39.830 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: ntpd: command not found 2024-04-20T02:41:39.843 INFO:teuthology.orchestra.run.smithi138.stdout:506 Cannot talk to daemon 2024-04-20T02:41:39.843 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-20T02:41:39.857 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-20T02:41:39.871 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-20T02:41:39.924 INFO:teuthology.orchestra.run.smithi105.stderr:bash: ntpq: command not found 2024-04-20T02:41:39.927 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T02:41:39.927 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-04-20T02:41:39.927 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:41:39.927 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:41:39.927 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:41:39.927 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:41:39.938 INFO:teuthology.orchestra.run.smithi138.stderr:bash: ntpq: command not found 2024-04-20T02:41:39.942 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T02:41:39.942 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-04-20T02:41:39.942 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:41:39.943 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:41:39.943 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:41:39.943 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:41:39.943 INFO:teuthology.run_tasks:Running task install... 2024-04-20T02:41:39.951 DEBUG:teuthology.task.install:project ceph 2024-04-20T02:41:39.951 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed'}} 2024-04-20T02:41:39.951 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed'} 2024-04-20T02:41:39.951 INFO:teuthology.task.install:Using flavor: default 2024-04-20T02:41:39.957 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-20T02:41:39.957 INFO:teuthology.task.install:extra packages: [] 2024-04-20T02:41:39.958 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed', 'tag': None, 'wait_for_package': False} 2024-04-20T02:41:39.958 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=15eecde2bab9803889d929c7e423d07970d8cbed 2024-04-20T02:41:39.960 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed', 'tag': None, 'wait_for_package': False} 2024-04-20T02:41:39.960 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=15eecde2bab9803889d929c7e423d07970d8cbed 2024-04-20T02:41:40.104 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-04-18-1351-reef/15eecde2bab9803889d929c7e423d07970d8cbed/centos/8/flavors/default/ 2024-04-20T02:41:40.104 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1252.g15eecde2 2024-04-20T02:41:40.178 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-04-18-1351-reef/15eecde2bab9803889d929c7e423d07970d8cbed/centos/8/flavors/default/ 2024-04-20T02:41:40.179 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1252.g15eecde2 2024-04-20T02:41:40.512 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-04-18-1351-reef/15eecde2bab9803889d929c7e423d07970d8cbed/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-04-18-1351-reef/15eecde2bab9803889d929c7e423d07970d8cbed/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-04-18-1351-reef/15eecde2bab9803889d929c7e423d07970d8cbed/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-20T02:41:40.512 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:41:40.512 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-20T02:41:40.514 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-04-18-1351-reef/15eecde2bab9803889d929c7e423d07970d8cbed/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-04-18-1351-reef/15eecde2bab9803889d929c7e423d07970d8cbed/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-04-18-1351-reef/15eecde2bab9803889d929c7e423d07970d8cbed/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-20T02:41:40.515 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:41:40.515 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-20T02:41:40.543 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-20T02:41:40.544 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/15eecde2bab9803889d929c7e423d07970d8cbed/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-20T02:41:40.545 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-20T02:41:40.545 DEBUG:teuthology.orchestra.run.smithi138:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/15eecde2bab9803889d929c7e423d07970d8cbed/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-20T02:41:40.613 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-20T02:41:40.616 DEBUG:teuthology.orchestra.run.smithi138:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-20T02:41:40.691 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-20T02:41:40.696 DEBUG:teuthology.orchestra.run.smithi138:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-20T02:41:40.758 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-04-20T02:41:40.759 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-04-20T02:41:40.763 INFO:teuthology.orchestra.run.smithi138.stdout:check_obsoletes = 1 2024-04-20T02:41:40.765 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum clean all 2024-04-20T02:41:41.134 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:41:41.140 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:41:42.400 INFO:teuthology.orchestra.run.smithi105.stdout:63 files removed 2024-04-20T02:41:42.449 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw 2024-04-20T02:41:42.506 INFO:teuthology.orchestra.run.smithi138.stdout:63 files removed 2024-04-20T02:41:42.559 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-radosgw 2024-04-20T02:41:42.783 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:41:42.894 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:41:44.516 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 184 kB/s | 80 kB 00:00 2024-04-20T02:41:44.694 INFO:teuthology.orchestra.run.smithi138.stdout:ceph packages for x86_64 173 kB/s | 80 kB 00:00 2024-04-20T02:41:44.900 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 33 kB/s | 11 kB 00:00 2024-04-20T02:41:45.087 INFO:teuthology.orchestra.run.smithi138.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-04-20T02:41:45.259 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2024-04-20T02:41:45.388 INFO:teuthology.orchestra.run.smithi138.stdout:ceph source packages 5.4 kB/s | 1.5 kB 00:00 2024-04-20T02:41:45.410 INFO:teuthology.orchestra.run.smithi105.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-04-20T02:41:45.538 INFO:teuthology.orchestra.run.smithi138.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-04-20T02:41:46.352 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2024-04-20T02:41:47.050 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-20T02:41:49.877 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 535 kB/s | 24 kB 00:00 2024-04-20T02:41:50.623 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 616 kB/s | 24 kB 00:00 2024-04-20T02:41:50.671 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 12 MB/s | 8.8 MB 00:00 2024-04-20T02:41:51.441 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 12 MB/s | 8.8 MB 00:00 2024-04-20T02:41:53.556 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 67 MB 00:01 2024-04-20T02:41:54.334 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 67 MB 00:01 2024-04-20T02:42:04.581 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 47 MB/s | 60 MB 00:01 2024-04-20T02:42:05.210 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 MB/s | 60 MB 00:01 2024-04-20T02:42:20.329 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout:==================================================================================================== 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout:==================================================================================================== 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 13 M 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 3.7 M 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 3.6 M 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 5.5 M 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 22 M 2024-04-20T02:42:20.332 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 24 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 781 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 474 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 6.5 M 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 45 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 129 k 2024-04-20T02:42:20.333 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 197 k 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 372 k 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 364 k 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 107 k 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout:==================================================================================================== 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout:Install 21 Packages 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 2 Packages 2024-04-20T02:42:20.334 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:42:20.335 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 59 M 2024-04-20T02:42:20.335 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:42:20.928 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:42:20.930 INFO:teuthology.orchestra.run.smithi138.stdout:==================================================================================================== 2024-04-20T02:42:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-20T02:42:20.931 INFO:teuthology.orchestra.run.smithi138.stdout:==================================================================================================== 2024-04-20T02:42:20.931 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:42:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 13 M 2024-04-20T02:42:20.931 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrading: 2024-04-20T02:42:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: librados2 x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 3.7 M 2024-04-20T02:42:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1 x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 3.6 M 2024-04-20T02:42:20.931 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 5.5 M 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 22 M 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 24 k 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2 x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 781 k 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1 x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 474 k 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2 x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 6.5 M 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-20T02:42:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 45 k 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 129 k 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 197 k 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 372 k 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 364 k 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 107 k 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout:==================================================================================================== 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout:Install 21 Packages 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrade 2 Packages 2024-04-20T02:42:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:42:20.934 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 59 M 2024-04-20T02:42:20.934 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:42:22.121 INFO:teuthology.orchestra.run.smithi105.stdout:(1/23): ceph-radosgw-18.2.2-1252.g15eecde2.el8. 9.6 MB/s | 13 MB 00:01 2024-04-20T02:42:22.190 INFO:teuthology.orchestra.run.smithi105.stdout:(2/23): ceph-selinux-18.2.2-1252.g15eecde2.el8. 359 kB/s | 24 kB 00:00 2024-04-20T02:42:22.323 INFO:teuthology.orchestra.run.smithi105.stdout:(3/23): libcephfs2-18.2.2-1252.g15eecde2.el8.x8 5.7 MB/s | 781 kB 00:00 2024-04-20T02:42:22.415 INFO:teuthology.orchestra.run.smithi105.stdout:(4/23): libradosstriper1-18.2.2-1252.g15eecde2. 5.1 MB/s | 474 kB 00:00 2024-04-20T02:42:22.649 INFO:teuthology.orchestra.run.smithi105.stdout:(5/23): ceph-common-18.2.2-1252.g15eecde2.el8.x 11 MB/s | 22 MB 00:01 2024-04-20T02:42:22.724 INFO:teuthology.orchestra.run.smithi105.stdout:(6/23): python3-ceph-argparse-18.2.2-1252.g15ee 597 kB/s | 45 kB 00:00 2024-04-20T02:42:22.799 INFO:teuthology.orchestra.run.smithi105.stdout:(7/23): python3-ceph-common-18.2.2-1252.g15eecd 1.7 MB/s | 129 kB 00:00 2024-04-20T02:42:22.891 INFO:teuthology.orchestra.run.smithi105.stdout:(8/23): python3-cephfs-18.2.2-1252.g15eecde2.el 2.1 MB/s | 197 kB 00:00 2024-04-20T02:42:22.975 INFO:teuthology.orchestra.run.smithi105.stdout:(9/23): librgw2-18.2.2-1252.g15eecde2.el8.x86_6 12 MB/s | 6.5 MB 00:00 2024-04-20T02:42:23.000 INFO:teuthology.orchestra.run.smithi105.stdout:(10/23): python3-rados-18.2.2-1252.g15eecde2.el 3.3 MB/s | 372 kB 00:00 2024-04-20T02:42:23.043 INFO:teuthology.orchestra.run.smithi105.stdout:(11/23): python3-rbd-18.2.2-1252.g15eecde2.el8. 5.3 MB/s | 364 kB 00:00 2024-04-20T02:42:23.094 INFO:teuthology.orchestra.run.smithi105.stdout:(12/23): python3-rgw-18.2.2-1252.g15eecde2.el8. 1.1 MB/s | 107 kB 00:00 2024-04-20T02:42:23.518 INFO:teuthology.orchestra.run.smithi105.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 139 kB/s | 59 kB 00:00 2024-04-20T02:42:23.677 INFO:teuthology.orchestra.run.smithi105.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 483 kB/s | 306 kB 00:00 2024-04-20T02:42:23.702 INFO:teuthology.orchestra.run.smithi105.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 409 kB/s | 75 kB 00:00 2024-04-20T02:42:23.713 INFO:teuthology.orchestra.run.smithi138.stdout:(1/23): ceph-base-18.2.2-1252.g15eecde2.el8.x86 2.1 MB/s | 5.5 MB 00:02 2024-04-20T02:42:23.759 INFO:teuthology.orchestra.run.smithi138.stdout:(2/23): ceph-selinux-18.2.2-1252.g15eecde2.el8. 542 kB/s | 24 kB 00:00 2024-04-20T02:42:23.969 INFO:teuthology.orchestra.run.smithi105.stdout:(16/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 177 kB/s | 47 kB 00:00 2024-04-20T02:42:24.017 INFO:teuthology.orchestra.run.smithi138.stdout:(3/23): ceph-common-18.2.2-1252.g15eecde2.el8.x 7.2 MB/s | 22 MB 00:02 2024-04-20T02:42:24.036 INFO:teuthology.orchestra.run.smithi105.stdout:(17/23): thrift-0.13.0-2.el8.x86_64.rpm 4.8 MB/s | 1.7 MB 00:00 2024-04-20T02:42:24.051 INFO:teuthology.orchestra.run.smithi138.stdout:(4/23): libcephfs2-18.2.2-1252.g15eecde2.el8.x8 2.6 MB/s | 781 kB 00:00 2024-04-20T02:42:24.076 INFO:teuthology.orchestra.run.smithi138.stdout:(5/23): libradosstriper1-18.2.2-1252.g15eecde2. 8.0 MB/s | 474 kB 00:00 2024-04-20T02:42:24.111 INFO:teuthology.orchestra.run.smithi105.stdout:(18/23): ceph-base-18.2.2-1252.g15eecde2.el8.x8 1.6 MB/s | 5.5 MB 00:03 2024-04-20T02:42:24.126 INFO:teuthology.orchestra.run.smithi138.stdout:(6/23): python3-ceph-argparse-18.2.2-1252.g15ee 894 kB/s | 45 kB 00:00 2024-04-20T02:42:24.137 INFO:teuthology.orchestra.run.smithi105.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 263 kB/s | 44 kB 00:00 2024-04-20T02:42:24.177 INFO:teuthology.orchestra.run.smithi138.stdout:(7/23): python3-ceph-common-18.2.2-1252.g15eecd 2.5 MB/s | 129 kB 00:00 2024-04-20T02:42:24.203 INFO:teuthology.orchestra.run.smithi105.stdout:(20/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2024-04-20T02:42:24.254 INFO:teuthology.orchestra.run.smithi105.stdout:(21/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.5 MB/s | 354 kB 00:00 2024-04-20T02:42:24.269 INFO:teuthology.orchestra.run.smithi138.stdout:(8/23): python3-cephfs-18.2.2-1252.g15eecde2.el 2.1 MB/s | 197 kB 00:00 2024-04-20T02:42:24.336 INFO:teuthology.orchestra.run.smithi138.stdout:(9/23): python3-rados-18.2.2-1252.g15eecde2.el8 5.5 MB/s | 372 kB 00:00 2024-04-20T02:42:24.436 INFO:teuthology.orchestra.run.smithi138.stdout:(10/23): python3-rbd-18.2.2-1252.g15eecde2.el8. 3.6 MB/s | 364 kB 00:00 2024-04-20T02:42:24.536 INFO:teuthology.orchestra.run.smithi138.stdout:(11/23): python3-rgw-18.2.2-1252.g15eecde2.el8. 1.0 MB/s | 107 kB 00:00 2024-04-20T02:42:24.654 INFO:teuthology.orchestra.run.smithi105.stdout:(22/23): librbd1-18.2.2-1252.g15eecde2.el8.x86_ 7.9 MB/s | 3.6 MB 00:00 2024-04-20T02:42:24.670 INFO:teuthology.orchestra.run.smithi138.stdout:(12/23): librgw2-18.2.2-1252.g15eecde2.el8.x86_ 11 MB/s | 6.5 MB 00:00 2024-04-20T02:42:24.954 INFO:teuthology.orchestra.run.smithi138.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 211 kB/s | 59 kB 00:00 2024-04-20T02:42:24.995 INFO:teuthology.orchestra.run.smithi138.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 667 kB/s | 306 kB 00:00 2024-04-20T02:42:25.046 INFO:teuthology.orchestra.run.smithi138.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 818 kB/s | 75 kB 00:00 2024-04-20T02:42:25.146 INFO:teuthology.orchestra.run.smithi138.stdout:(16/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 471 kB/s | 47 kB 00:00 2024-04-20T02:42:25.196 INFO:teuthology.orchestra.run.smithi138.stdout:(17/23): thrift-0.13.0-2.el8.x86_64.rpm 8.6 MB/s | 1.7 MB 00:00 2024-04-20T02:42:25.313 INFO:teuthology.orchestra.run.smithi138.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.2 MB/s | 259 kB 00:00 2024-04-20T02:42:25.389 INFO:teuthology.orchestra.run.smithi138.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 181 kB/s | 44 kB 00:00 2024-04-20T02:42:25.456 INFO:teuthology.orchestra.run.smithi138.stdout:(20/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2024-04-20T02:42:25.479 INFO:teuthology.orchestra.run.smithi105.stdout:(23/23): librados2-18.2.2-1252.g15eecde2.el8.x8 2.8 MB/s | 3.7 MB 00:01 2024-04-20T02:42:25.483 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:42:25.484 INFO:teuthology.orchestra.run.smithi105.stdout:Total 12 MB/s | 59 MB 00:05 2024-04-20T02:42:25.533 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:42:25.580 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:42:25.580 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:42:25.773 INFO:teuthology.orchestra.run.smithi138.stdout:(21/23): librados2-18.2.2-1252.g15eecde2.el8.x8 9.6 MB/s | 3.7 MB 00:00 2024-04-20T02:42:25.848 INFO:teuthology.orchestra.run.smithi138.stdout:(22/23): librbd1-18.2.2-1252.g15eecde2.el8.x86_ 9.1 MB/s | 3.6 MB 00:00 2024-04-20T02:42:25.911 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:42:25.920 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:42:26.173 INFO:teuthology.orchestra.run.smithi138.stdout:(23/23): ceph-radosgw-18.2.2-1252.g15eecde2.el8 2.5 MB/s | 13 MB 00:05 2024-04-20T02:42:26.177 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:42:26.178 INFO:teuthology.orchestra.run.smithi138.stdout:Total 11 MB/s | 59 MB 00:05 2024-04-20T02:42:26.224 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:42:26.270 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:42:26.270 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:42:26.600 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:42:26.610 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:42:27.213 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:42:27.437 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-20T02:42:27.612 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-20T02:42:27.893 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:42:28.023 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-20T02:42:28.122 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-20T02:42:28.279 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:18.2.2-1252.g15eecde2.el8.x86_64 3/25 2024-04-20T02:42:28.305 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-20T02:42:28.379 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:18.2.2-1252.g15eecde2.el8.x86_64 3/25 2024-04-20T02:42:28.559 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:18.2.2-1252.g15eecde2.el8.x86_64 4/25 2024-04-20T02:42:28.701 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-20T02:42:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-20T02:42:28.956 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librados2-2:18.2.2-1252.g15eecde2.el8.x86_64 3/25 2024-04-20T02:42:29.029 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:18.2.2-1252.g15eecde2.el8.x86_64 6/25 2024-04-20T02:42:29.050 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:18.2.2-1252.g15eecde2.el8.x86_64 3/25 2024-04-20T02:42:29.244 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rados-2:18.2.2-1252.g15eecde2.el8.x86_64 4/25 2024-04-20T02:42:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:18.2.2-1252.g15eecde2.el8.x86_64 6/25 2024-04-20T02:42:29.529 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-20T02:42:29.747 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs2-2:18.2.2-1252.g15eecde2.el8.x86_64 6/25 2024-04-20T02:42:29.856 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:18.2.2-1252.g15eecde2.el8.x86_64 7/25 2024-04-20T02:42:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:18.2.2-1252.g15eecde2.el8.x86_64 7/25 2024-04-20T02:42:30.139 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs2-2:18.2.2-1252.g15eecde2.el8.x86_64 6/25 2024-04-20T02:42:30.173 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-20T02:42:30.228 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-20T02:42:30.330 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librbd1-2:18.2.2-1252.g15eecde2.el8.x86_64 7/25 2024-04-20T02:42:30.407 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:18.2.2-1252.g15eecde2.el8.x86_64 7/25 2024-04-20T02:42:30.443 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 9/25 2024-04-20T02:42:30.573 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:18.2.2-1252.g15eecde2.el8. 10/25 2024-04-20T02:42:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-20T02:42:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-20T02:42:30.815 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:18.2.2-1252.g15eecde2.el8.x86_64 11/25 2024-04-20T02:42:30.986 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 9/25 2024-04-20T02:42:31.055 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:18.2.2-1252.g15eecde2.el8.x86_64 12/25 2024-04-20T02:42:31.132 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-argparse-2:18.2.2-1252.g15eecde2.el8. 10/25 2024-04-20T02:42:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:18.2.2-1252.g15eecde2.el8.x86_6 13/25 2024-04-20T02:42:31.353 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1252.g15eecde2.el8.x86_6 13/25 2024-04-20T02:42:31.365 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cephfs-2:18.2.2-1252.g15eecde2.el8.x86_64 11/25 2024-04-20T02:42:31.488 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-20T02:42:31.638 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rbd-2:18.2.2-1252.g15eecde2.el8.x86_64 12/25 2024-04-20T02:42:31.713 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-20T02:42:31.871 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libradosstriper1-2:18.2.2-1252.g15eecde2.el8.x86_6 13/25 2024-04-20T02:42:31.938 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1252.g15eecde2.el8.x86_6 13/25 2024-04-20T02:42:32.130 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-20T02:42:32.363 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-20T02:42:32.538 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-20T02:42:32.711 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:18.2.2-1252.g15eecde2.el8.x86_64 17/25 2024-04-20T02:42:32.815 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:18.2.2-1252.g15eecde2.el8.x86_64 17/25 2024-04-20T02:42:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:18.2.2-1252.g15eecde2.el8.x86_64 18/25 2024-04-20T02:42:33.188 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-20T02:42:33.295 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:18.2.2-1252.g15eecde2.el8.x8 19/25 2024-04-20T02:42:33.404 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librgw2-2:18.2.2-1252.g15eecde2.el8.x86_64 17/25 2024-04-20T02:42:33.478 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librgw2-2:18.2.2-1252.g15eecde2.el8.x86_64 17/25 2024-04-20T02:42:33.760 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rgw-2:18.2.2-1252.g15eecde2.el8.x86_64 18/25 2024-04-20T02:42:33.930 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-common-2:18.2.2-1252.g15eecde2.el8.x8 19/25 2024-04-20T02:42:37.025 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 20/25 2024-04-20T02:42:37.295 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 20/25 2024-04-20T02:42:37.713 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 20/25 2024-04-20T02:42:37.886 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 20/25 2024-04-20T02:42:37.962 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 20/25 2024-04-20T02:42:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:18.2.2-1252.g15eecde2.el8.x86_64 21/25 2024-04-20T02:42:38.284 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:18.2.2-1252.g15eecde2.el8.x86_64 21/25 2024-04-20T02:42:38.448 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:18.2.2-1252.g15eecde2.el8.x86_64 22/25 2024-04-20T02:42:38.554 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 20/25 2024-04-20T02:42:38.868 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-base-2:18.2.2-1252.g15eecde2.el8.x86_64 21/25 2024-04-20T02:42:38.958 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-base-2:18.2.2-1252.g15eecde2.el8.x86_64 21/25 2024-04-20T02:42:39.178 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-selinux-2:18.2.2-1252.g15eecde2.el8.x86_64 22/25 2024-04-20T02:42:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1252.g15eecde2.el8.x86_64 22/25 2024-04-20T02:42:57.926 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-04-20T02:42:57.926 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-04-20T02:42:57.926 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /dev 2024-04-20T02:42:57.926 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /run 2024-04-20T02:42:57.927 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-04-20T02:42:57.927 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-04-20T02:42:57.927 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-04-20T02:42:57.927 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-04-20T02:42:57.927 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /dev 2024-04-20T02:42:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:42:58.567 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1252.g15eecde2.el8.x86_64 22/25 2024-04-20T02:42:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /sys 2024-04-20T02:42:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /proc 2024-04-20T02:42:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /dev 2024-04-20T02:42:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /run 2024-04-20T02:42:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /mnt 2024-04-20T02:42:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /var/tmp 2024-04-20T02:42:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /home 2024-04-20T02:42:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /tmp 2024-04-20T02:42:58.568 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /dev 2024-04-20T02:42:58.568 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:42:59.269 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:18.2.2-1252.g15eecde2.el8.x86_64 23/25 2024-04-20T02:42:59.291 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1252.g15eecde2.el8.x86_64 23/25 2024-04-20T02:42:59.291 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-20T02:42:59.503 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-20T02:42:59.503 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T02:42:59.922 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-radosgw-2:18.2.2-1252.g15eecde2.el8.x86_64 23/25 2024-04-20T02:42:59.944 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1252.g15eecde2.el8.x86_64 23/25 2024-04-20T02:42:59.944 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-20T02:43:00.186 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-20T02:43:00.186 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T02:43:00.230 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T02:43:00.230 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:18.2.2-1252.g15eecde2.el8.x86_64 1/25 2024-04-20T02:43:00.230 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 2/25 2024-04-20T02:43:00.230 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:18.2.2-1252.g15eecde2.el8.x86_64 3/25 2024-04-20T02:43:00.230 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:18.2.2-1252.g15eecde2.el8.x86_64 4/25 2024-04-20T02:43:00.231 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:18.2.2-1252.g15eecde2.el8.x86_64 5/25 2024-04-20T02:43:00.231 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:18.2.2-1252.g15eecde2.el8.x86_6 6/25 2024-04-20T02:43:00.231 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:18.2.2-1252.g15eecde2.el8.x86_64 7/25 2024-04-20T02:43:00.231 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1252.g15eecde2.el8. 8/25 2024-04-20T02:43:00.231 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:18.2.2-1252.g15eecde2.el8.x8 9/25 2024-04-20T02:43:00.231 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:18.2.2-1252.g15eecde2.el8.x86_64 10/25 2024-04-20T02:43:00.231 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:18.2.2-1252.g15eecde2.el8.x86_64 11/25 2024-04-20T02:43:00.232 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:18.2.2-1252.g15eecde2.el8.x86_64 12/25 2024-04-20T02:43:00.232 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:18.2.2-1252.g15eecde2.el8.x86_64 13/25 2024-04-20T02:43:00.232 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-20T02:43:00.232 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-20T02:43:00.232 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-20T02:43:00.232 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-20T02:43:00.233 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 18/25 2024-04-20T02:43:00.233 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2024-04-20T02:43:00.233 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2024-04-20T02:43:00.233 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2024-04-20T02:43:00.233 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:18.2.2-1252.g15eecde2.el8.x86_64 22/25 2024-04-20T02:43:00.233 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-20T02:43:00.233 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:18.2.2-1252.g15eecde2.el8.x86_64 24/25 2024-04-20T02:43:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T02:43:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-base-2:18.2.2-1252.g15eecde2.el8.x86_64 1/25 2024-04-20T02:43:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 2/25 2024-04-20T02:43:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-radosgw-2:18.2.2-1252.g15eecde2.el8.x86_64 3/25 2024-04-20T02:43:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-selinux-2:18.2.2-1252.g15eecde2.el8.x86_64 4/25 2024-04-20T02:43:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs2-2:18.2.2-1252.g15eecde2.el8.x86_64 5/25 2024-04-20T02:43:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libradosstriper1-2:18.2.2-1252.g15eecde2.el8.x86_6 6/25 2024-04-20T02:43:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librgw2-2:18.2.2-1252.g15eecde2.el8.x86_64 7/25 2024-04-20T02:43:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1252.g15eecde2.el8. 8/25 2024-04-20T02:43:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-common-2:18.2.2-1252.g15eecde2.el8.x8 9/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cephfs-2:18.2.2-1252.g15eecde2.el8.x86_64 10/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rados-2:18.2.2-1252.g15eecde2.el8.x86_64 11/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rbd-2:18.2.2-1252.g15eecde2.el8.x86_64 12/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rgw-2:18.2.2-1252.g15eecde2.el8.x86_64 13/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 18/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:18.2.2-1252.g15eecde2.el8.x86_64 22/25 2024-04-20T02:43:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-20T02:43:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:18.2.2-1252.g15eecde2.el8.x86_64 24/25 2024-04-20T02:43:08.705 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T02:43:08.705 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:43:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-20T02:43:08.889 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:43:09.262 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-20T02:43:09.263 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:43:09.264 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout:Upgraded: 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: librados2-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-20T02:43:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:43:09.439 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:43:09.474 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-test 2024-04-20T02:43:09.637 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-test 2024-04-20T02:43:09.830 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:43:09.991 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:43:11.513 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 47 kB/s | 4.1 kB 00:00 2024-04-20T02:43:11.898 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-20T02:43:12.002 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 19 kB/s | 4.1 kB 00:00 2024-04-20T02:43:12.390 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-20T02:43:14.251 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================================ 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================================ 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 47 M 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 142 k 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-20T02:43:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:43:14.253 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:43:14.253 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================================ 2024-04-20T02:43:14.253 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2024-04-20T02:43:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:43:14.253 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 48 M 2024-04-20T02:43:14.253 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 213 M 2024-04-20T02:43:14.254 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:43:14.293 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 70 kB 00:00 2024-04-20T02:43:14.511 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.5 MB/s | 323 kB 00:00 2024-04-20T02:43:14.544 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): libcephsqlite-18.2.2-1252.g15eecde2.el8. 489 kB/s | 142 kB 00:00 2024-04-20T02:43:14.611 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): jq-1.6-7.el8.x86_64.rpm 2.0 MB/s | 202 kB 00:00 2024-04-20T02:43:14.752 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:43:14.753 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-04-20T02:43:14.753 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-20T02:43:14.753 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 47 M 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 142 k 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================================ 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2024-04-20T02:43:14.754 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:43:14.755 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 48 M 2024-04-20T02:43:14.755 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 213 M 2024-04-20T02:43:14.755 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:43:14.778 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 806 kB/s | 188 kB 00:00 2024-04-20T02:43:14.791 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.9 MB/s | 70 kB 00:00 2024-04-20T02:43:14.995 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.6 MB/s | 323 kB 00:00 2024-04-20T02:43:15.142 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 1.3 MB/s | 202 kB 00:00 2024-04-20T02:43:15.334 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 983 kB/s | 188 kB 00:00 2024-04-20T02:43:15.534 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): libcephsqlite-18.2.2-1252.g15eecde2.el8. 182 kB/s | 142 kB 00:00 2024-04-20T02:43:16.795 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): ceph-test-18.2.2-1252.g15eecde2.el8.x86_ 19 MB/s | 47 MB 00:02 2024-04-20T02:43:16.796 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:43:16.796 INFO:teuthology.orchestra.run.smithi105.stdout:Total 19 MB/s | 48 MB 00:02 2024-04-20T02:43:16.834 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:43:16.855 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:43:16.856 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:43:17.124 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:43:17.132 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:43:17.630 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:43:17.814 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-20T02:43:17.898 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-20T02:43:18.151 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-20T02:43:18.342 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): ceph-test-18.2.2-1252.g15eecde2.el8.x86_ 13 MB/s | 47 MB 00:03 2024-04-20T02:43:18.345 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:43:18.345 INFO:teuthology.orchestra.run.smithi138.stdout:Total 13 MB/s | 48 MB 00:03 2024-04-20T02:43:18.347 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-20T02:43:18.384 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:43:18.405 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:43:18.405 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:43:18.551 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-20T02:43:18.673 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:43:18.682 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:43:18.755 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:18.2.2-1252.g15eecde2.el8.x86_64 5/6 2024-04-20T02:43:19.168 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:43:19.345 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-20T02:43:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-20T02:43:19.609 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-20T02:43:19.781 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-20T02:43:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-20T02:43:20.205 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephsqlite-2:18.2.2-1252.g15eecde2.el8.x86_64 5/6 2024-04-20T02:43:23.279 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1252.g15eecde2.el8.x86_64 5/6 2024-04-20T02:43:23.552 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:18.2.2-1252.g15eecde2.el8.x86_64 6/6 2024-04-20T02:43:23.796 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-test-2:18.2.2-1252.g15eecde2.el8.x86_64 6/6 2024-04-20T02:43:23.796 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:18.2.2-1252.g15eecde2.el8.x86_64 1/6 2024-04-20T02:43:23.796 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:18.2.2-1252.g15eecde2.el8.x86_64 2/6 2024-04-20T02:43:23.796 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-20T02:43:23.796 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-20T02:43:23.796 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-20T02:43:24.691 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1252.g15eecde2.el8.x86_64 5/6 2024-04-20T02:43:24.961 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-test-2:18.2.2-1252.g15eecde2.el8.x86_64 6/6 2024-04-20T02:43:25.227 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-test-2:18.2.2-1252.g15eecde2.el8.x86_64 6/6 2024-04-20T02:43:25.227 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-test-2:18.2.2-1252.g15eecde2.el8.x86_64 1/6 2024-04-20T02:43:25.227 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephsqlite-2:18.2.2-1252.g15eecde2.el8.x86_64 2/6 2024-04-20T02:43:25.227 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-20T02:43:25.227 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-20T02:43:25.227 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-20T02:43:36.293 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-20T02:43:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:43:36.681 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-20T02:43:36.681 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:43:36.846 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:43:36.846 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:43:36.846 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:36.847 INFO:teuthology.orchestra.run.smithi105.stdout: jq-1.6-7.el8.x86_64 2024-04-20T02:43:36.847 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:36.847 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-20T02:43:36.847 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-20T02:43:36.847 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-20T02:43:36.847 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:43:36.847 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:43:37.004 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph 2024-04-20T02:43:37.230 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:43:37.230 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:43:37.230 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:37.230 INFO:teuthology.orchestra.run.smithi138.stdout: jq-1.6-7.el8.x86_64 2024-04-20T02:43:37.230 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:43:37.230 INFO:teuthology.orchestra.run.smithi138.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-20T02:43:37.230 INFO:teuthology.orchestra.run.smithi138.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-20T02:43:37.230 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-20T02:43:37.230 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:43:37.230 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:43:37.343 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:43:37.403 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph 2024-04-20T02:43:37.739 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:43:39.324 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-20T02:43:39.613 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 40 kB/s | 4.1 kB 00:00 2024-04-20T02:43:39.665 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-20T02:43:39.935 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-04-20T02:43:42.055 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:43:42.058 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================= 2024-04-20T02:43:42.058 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================= 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 6.4 k 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 2.3 M 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 1.6 M 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 246 k 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 4.8 M 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 18 M 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-20T02:43:42.059 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-20T02:43:42.060 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================= 2024-04-20T02:43:42.061 INFO:teuthology.orchestra.run.smithi105.stdout:Install 32 Packages 2024-04-20T02:43:42.062 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:43:42.062 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 31 M 2024-04-20T02:43:42.062 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 124 M 2024-04-20T02:43:42.062 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:43:42.286 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:43:42.289 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================= 2024-04-20T02:43:42.289 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-20T02:43:42.289 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================= 2024-04-20T02:43:42.289 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:43:42.289 INFO:teuthology.orchestra.run.smithi138.stdout: ceph x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 6.4 k 2024-04-20T02:43:42.289 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 2.3 M 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 1.6 M 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 246 k 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 4.8 M 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 18 M 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-20T02:43:42.290 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-20T02:43:42.291 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-20T02:43:42.292 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-20T02:43:42.292 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-20T02:43:42.292 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:43:42.292 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:43:42.292 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================= 2024-04-20T02:43:42.292 INFO:teuthology.orchestra.run.smithi138.stdout:Install 32 Packages 2024-04-20T02:43:42.292 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:43:42.292 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 31 M 2024-04-20T02:43:42.293 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 124 M 2024-04-20T02:43:42.293 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:43:42.540 INFO:teuthology.orchestra.run.smithi105.stdout:(1/32): ceph-18.2.2-1252.g15eecde2.el8.x86_64.r 45 kB/s | 6.4 kB 00:00 2024-04-20T02:43:42.682 INFO:teuthology.orchestra.run.smithi105.stdout:(2/32): ceph-mgr-18.2.2-1252.g15eecde2.el8.x86_ 5.5 MB/s | 1.6 MB 00:00 2024-04-20T02:43:42.902 INFO:teuthology.orchestra.run.smithi138.stdout:(1/32): ceph-mds-18.2.2-1252.g15eecde2.el8.x86_ 5.4 MB/s | 2.3 MB 00:00 2024-04-20T02:43:42.987 INFO:teuthology.orchestra.run.smithi138.stdout:(2/32): ceph-18.2.2-1252.g15eecde2.el8.x86_64.r 13 kB/s | 6.4 kB 00:00 2024-04-20T02:43:43.258 INFO:teuthology.orchestra.run.smithi105.stdout:(3/32): ceph-mon-18.2.2-1252.g15eecde2.el8.x86_ 6.7 MB/s | 4.8 MB 00:00 2024-04-20T02:43:43.341 INFO:teuthology.orchestra.run.smithi105.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1252.g15ee 2.9 MB/s | 246 kB 00:00 2024-04-20T02:43:43.471 INFO:teuthology.orchestra.run.smithi138.stdout:(3/32): ceph-mon-18.2.2-1252.g15eecde2.el8.x86_ 8.5 MB/s | 4.8 MB 00:00 2024-04-20T02:43:43.538 INFO:teuthology.orchestra.run.smithi138.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1252.g15ee 3.6 MB/s | 246 kB 00:00 2024-04-20T02:43:43.680 INFO:teuthology.orchestra.run.smithi138.stdout:(5/32): ceph-mgr-18.2.2-1252.g15eecde2.el8.x86_ 1.3 MB/s | 1.6 MB 00:01 2024-04-20T02:43:43.716 INFO:teuthology.orchestra.run.smithi105.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 116 kB/s | 44 kB 00:00 2024-04-20T02:43:43.922 INFO:teuthology.orchestra.run.smithi138.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 114 kB/s | 44 kB 00:00 2024-04-20T02:43:44.134 INFO:teuthology.orchestra.run.smithi105.stdout:(6/32): ceph-osd-18.2.2-1252.g15eecde2.el8.x86_ 13 MB/s | 18 MB 00:01 2024-04-20T02:43:44.159 INFO:teuthology.orchestra.run.smithi105.stdout:(7/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 419 kB/s | 185 kB 00:00 2024-04-20T02:43:44.197 INFO:teuthology.orchestra.run.smithi138.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 628 kB/s | 173 kB 00:00 2024-04-20T02:43:44.256 INFO:teuthology.orchestra.run.smithi138.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 322 kB/s | 185 kB 00:00 2024-04-20T02:43:44.293 INFO:teuthology.orchestra.run.smithi105.stdout:(8/32): ceph-mds-18.2.2-1252.g15eecde2.el8.x86_ 1.2 MB/s | 2.3 MB 00:01 2024-04-20T02:43:44.348 INFO:teuthology.orchestra.run.smithi138.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-04-20T02:43:44.376 INFO:teuthology.orchestra.run.smithi105.stdout:(9/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 1.7 MB/s | 384 kB 00:00 2024-04-20T02:43:44.415 INFO:teuthology.orchestra.run.smithi138.stdout:(10/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2024-04-20T02:43:44.448 INFO:teuthology.orchestra.run.smithi138.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-04-20T02:43:44.485 INFO:teuthology.orchestra.run.smithi105.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 169 kB/s | 18 kB 00:00 2024-04-20T02:43:44.524 INFO:teuthology.orchestra.run.smithi138.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 448 kB/s | 49 kB 00:00 2024-04-20T02:43:44.565 INFO:teuthology.orchestra.run.smithi138.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 501 kB/s | 59 kB 00:00 2024-04-20T02:43:44.593 INFO:teuthology.orchestra.run.smithi105.stdout:(11/32): python3-jaraco-6.2-6.el8.noarch.rpm 37 kB/s | 11 kB 00:00 2024-04-20T02:43:44.619 INFO:teuthology.orchestra.run.smithi105.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 365 kB/s | 49 kB 00:00 2024-04-20T02:43:44.658 INFO:teuthology.orchestra.run.smithi138.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2024-04-20T02:43:44.660 INFO:teuthology.orchestra.run.smithi105.stdout:(13/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 328 kB/s | 173 kB 00:00 2024-04-20T02:43:44.683 INFO:teuthology.orchestra.run.smithi138.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2024-04-20T02:43:44.744 INFO:teuthology.orchestra.run.smithi105.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.2 MB/s | 283 kB 00:00 2024-04-20T02:43:44.783 INFO:teuthology.orchestra.run.smithi138.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 149 kB/s | 19 kB 00:00 2024-04-20T02:43:44.795 INFO:teuthology.orchestra.run.smithi105.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 115 kB/s | 16 kB 00:00 2024-04-20T02:43:44.808 INFO:teuthology.orchestra.run.smithi138.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 191 kB/s | 24 kB 00:00 2024-04-20T02:43:44.828 INFO:teuthology.orchestra.run.smithi105.stdout:(16/32): python3-more-itertools-7.2.0-3.el8.noa 250 kB/s | 59 kB 00:00 2024-04-20T02:43:44.853 INFO:teuthology.orchestra.run.smithi105.stdout:(17/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2024-04-20T02:43:44.884 INFO:teuthology.orchestra.run.smithi138.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 292 kB/s | 29 kB 00:00 2024-04-20T02:43:44.895 INFO:teuthology.orchestra.run.smithi105.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 241 kB/s | 24 kB 00:00 2024-04-20T02:43:44.917 INFO:teuthology.orchestra.run.smithi138.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2024-04-20T02:43:44.946 INFO:teuthology.orchestra.run.smithi105.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 249 kB/s | 29 kB 00:00 2024-04-20T02:43:44.963 INFO:teuthology.orchestra.run.smithi105.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 252 kB/s | 27 kB 00:00 2024-04-20T02:43:45.071 INFO:teuthology.orchestra.run.smithi105.stdout:(21/32): python3-webtest-2.0.33-1.el8.noarch.rp 787 kB/s | 85 kB 00:00 2024-04-20T02:43:45.097 INFO:teuthology.orchestra.run.smithi105.stdout:(22/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.2 MB/s | 255 kB 00:00 2024-04-20T02:43:45.126 INFO:teuthology.orchestra.run.smithi138.stdout:(20/32): ceph-osd-18.2.2-1252.g15eecde2.el8.x86 8.6 MB/s | 18 MB 00:02 2024-04-20T02:43:45.151 INFO:teuthology.orchestra.run.smithi138.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 956 kB/s | 255 kB 00:00 2024-04-20T02:43:45.163 INFO:teuthology.orchestra.run.smithi105.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2024-04-20T02:43:45.168 INFO:teuthology.orchestra.run.smithi138.stdout:(22/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.0 MB/s | 251 kB 00:00 2024-04-20T02:43:45.222 INFO:teuthology.orchestra.run.smithi105.stdout:(24/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 910 kB/s | 251 kB 00:00 2024-04-20T02:43:45.269 INFO:teuthology.orchestra.run.smithi138.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-04-20T02:43:45.314 INFO:teuthology.orchestra.run.smithi105.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 503 kB/s | 109 kB 00:00 2024-04-20T02:43:45.394 INFO:teuthology.orchestra.run.smithi138.stdout:(24/32): python3-pycparser-2.14-14.el8.noarch.r 485 kB/s | 109 kB 00:00 2024-04-20T02:43:45.431 INFO:teuthology.orchestra.run.smithi105.stdout:(26/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 890 kB/s | 238 kB 00:00 2024-04-20T02:43:45.456 INFO:teuthology.orchestra.run.smithi105.stdout:(27/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.2 MB/s | 457 kB 00:00 2024-04-20T02:43:45.477 INFO:teuthology.orchestra.run.smithi138.stdout:(25/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2024-04-20T02:43:45.569 INFO:teuthology.orchestra.run.smithi138.stdout:(26/32): python3-webtest-2.0.33-1.el8.noarch.rp 193 kB/s | 85 kB 00:00 2024-04-20T02:43:45.632 INFO:teuthology.orchestra.run.smithi105.stdout:(28/32): python3-cryptography-3.2.1-7.el8_9.x86 1.3 MB/s | 559 kB 00:00 2024-04-20T02:43:45.682 INFO:teuthology.orchestra.run.smithi105.stdout:(29/32): python3-markupsafe-0.23-19.el8.x86_64. 155 kB/s | 39 kB 00:00 2024-04-20T02:43:45.712 INFO:teuthology.orchestra.run.smithi138.stdout:(27/32): python3-markupsafe-0.23-19.el8.x86_64. 273 kB/s | 39 kB 00:00 2024-04-20T02:43:45.774 INFO:teuthology.orchestra.run.smithi105.stdout:(30/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 326 kB/s | 103 kB 00:00 2024-04-20T02:43:45.799 INFO:teuthology.orchestra.run.smithi105.stdout:(31/32): python3-mako-1.0.6-14.el8.noarch.rpm 940 kB/s | 157 kB 00:00 2024-04-20T02:43:45.828 INFO:teuthology.orchestra.run.smithi138.stdout:(28/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.3 MB/s | 457 kB 00:00 2024-04-20T02:43:45.912 INFO:teuthology.orchestra.run.smithi138.stdout:(29/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.8 MB/s | 157 kB 00:00 2024-04-20T02:43:45.946 INFO:teuthology.orchestra.run.smithi138.stdout:(30/32): python3-cryptography-3.2.1-7.el8_9.x86 1.0 MB/s | 559 kB 00:00 2024-04-20T02:43:45.982 INFO:teuthology.orchestra.run.smithi105.stdout:(32/32): python3-pytz-2017.2-11.el8.noarch.rpm 181 kB/s | 54 kB 00:00 2024-04-20T02:43:45.986 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:43:45.986 INFO:teuthology.orchestra.run.smithi105.stdout:Total 7.9 MB/s | 31 MB 00:03 2024-04-20T02:43:46.038 INFO:teuthology.orchestra.run.smithi138.stdout:(31/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 318 kB/s | 103 kB 00:00 2024-04-20T02:43:46.083 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:43:46.120 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:43:46.120 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:43:46.160 INFO:teuthology.orchestra.run.smithi138.stdout:(32/32): python3-pytz-2017.2-11.el8.noarch.rpm 219 kB/s | 54 kB 00:00 2024-04-20T02:43:46.164 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:43:46.164 INFO:teuthology.orchestra.run.smithi138.stdout:Total 8.0 MB/s | 31 MB 00:03 2024-04-20T02:43:46.259 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:43:46.296 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:43:46.296 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:43:46.349 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:43:46.358 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:43:46.522 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:43:46.530 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:43:47.595 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:43:47.733 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:43:47.806 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-20T02:43:47.957 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-20T02:43:47.990 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-20T02:43:48.163 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2024-04-20T02:43:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-20T02:43:48.362 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2024-04-20T02:43:48.394 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2024-04-20T02:43:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2024-04-20T02:43:48.770 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2024-04-20T02:43:48.916 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2024-04-20T02:43:49.013 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2024-04-20T02:43:49.244 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2024-04-20T02:43:49.472 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2024-04-20T02:43:49.649 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2024-04-20T02:43:49.678 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2024-04-20T02:43:49.978 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 9/32 2024-04-20T02:43:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2024-04-20T02:43:50.175 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2024-04-20T02:43:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2024-04-20T02:43:50.412 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 9/32 2024-04-20T02:43:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2024-04-20T02:43:50.626 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2024-04-20T02:43:50.810 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2024-04-20T02:43:50.847 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-20T02:43:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2024-04-20T02:43:51.108 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-20T02:43:51.287 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-20T02:43:51.299 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-20T02:43:51.471 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-20T02:43:51.498 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-20T02:43:51.688 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-20T02:43:51.688 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-20T02:43:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-20T02:43:51.895 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-20T02:43:52.112 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-20T02:43:52.121 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-20T02:43:52.248 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-20T02:43:52.355 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-20T02:43:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-20T02:43:52.573 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-20T02:43:52.704 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-20T02:43:52.854 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-20T02:43:52.923 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-20T02:43:53.123 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-20T02:43:53.181 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-20T02:43:53.369 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-20T02:43:53.423 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-20T02:43:53.644 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-20T02:43:53.968 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-20T02:43:54.344 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-20T02:43:54.435 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-20T02:43:54.770 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1252.g15eecde2.el8. 27/32 2024-04-20T02:43:55.942 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-20T02:43:56.368 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1252.g15eecde2.el8. 27/32 2024-04-20T02:43:56.545 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:18.2.2-1252.g15eecde2.el8.x86_64 28/32 2024-04-20T02:43:57.139 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-2:18.2.2-1252.g15eecde2.el8.x86_64 28/32 2024-04-20T02:43:58.183 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1252.g15eecde2.el8.x86_64 28/32 2024-04-20T02:43:58.453 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:18.2.2-1252.g15eecde2.el8.x86_64 29/32 2024-04-20T02:43:58.781 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1252.g15eecde2.el8.x86_64 28/32 2024-04-20T02:43:58.932 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:18.2.2-1252.g15eecde2.el8.x86_64 29/32 2024-04-20T02:43:59.046 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-osd-2:18.2.2-1252.g15eecde2.el8.x86_64 29/32 2024-04-20T02:43:59.123 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:18.2.2-1252.g15eecde2.el8.x86_64 30/32 2024-04-20T02:43:59.337 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:18.2.2-1252.g15eecde2.el8.x86_64 30/32 2024-04-20T02:43:59.509 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:18.2.2-1252.g15eecde2.el8.x86_64 31/32 2024-04-20T02:43:59.524 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-osd-2:18.2.2-1252.g15eecde2.el8.x86_64 29/32 2024-04-20T02:43:59.527 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:18.2.2-1252.g15eecde2.el8.x86_64 31/32 2024-04-20T02:43:59.674 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:18.2.2-1252.g15eecde2.el8.x86_64 32/32 2024-04-20T02:43:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mon-2:18.2.2-1252.g15eecde2.el8.x86_64 30/32 2024-04-20T02:43:59.940 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mon-2:18.2.2-1252.g15eecde2.el8.x86_64 30/32 2024-04-20T02:44:00.071 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-2:18.2.2-1252.g15eecde2.el8.x86_64 32/32 2024-04-20T02:44:00.071 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:18.2.2-1252.g15eecde2.el8.x86_64 1/32 2024-04-20T02:44:00.071 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:18.2.2-1252.g15eecde2.el8.x86_64 2/32 2024-04-20T02:44:00.071 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:18.2.2-1252.g15eecde2.el8.x86_64 3/32 2024-04-20T02:44:00.071 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:18.2.2-1252.g15eecde2.el8.x86_64 4/32 2024-04-20T02:44:00.071 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:18.2.2-1252.g15eecde2.el8.x86_64 5/32 2024-04-20T02:44:00.071 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1252.g15eecde2.el8. 6/32 2024-04-20T02:44:00.071 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-20T02:44:00.071 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-20T02:44:00.072 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-20T02:44:00.073 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-20T02:44:00.073 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-20T02:44:00.073 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2024-04-20T02:44:00.073 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2024-04-20T02:44:00.073 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 27/32 2024-04-20T02:44:00.073 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2024-04-20T02:44:00.073 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2024-04-20T02:44:00.073 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2024-04-20T02:44:00.073 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2024-04-20T02:44:00.124 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mds-2:18.2.2-1252.g15eecde2.el8.x86_64 31/32 2024-04-20T02:44:00.143 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mds-2:18.2.2-1252.g15eecde2.el8.x86_64 31/32 2024-04-20T02:44:00.281 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-2:18.2.2-1252.g15eecde2.el8.x86_64 32/32 2024-04-20T02:44:00.681 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-2:18.2.2-1252.g15eecde2.el8.x86_64 32/32 2024-04-20T02:44:00.681 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-2:18.2.2-1252.g15eecde2.el8.x86_64 1/32 2024-04-20T02:44:00.681 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mds-2:18.2.2-1252.g15eecde2.el8.x86_64 2/32 2024-04-20T02:44:00.681 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-2:18.2.2-1252.g15eecde2.el8.x86_64 3/32 2024-04-20T02:44:00.681 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mon-2:18.2.2-1252.g15eecde2.el8.x86_64 4/32 2024-04-20T02:44:00.681 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-osd-2:18.2.2-1252.g15eecde2.el8.x86_64 5/32 2024-04-20T02:44:00.681 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1252.g15eecde2.el8. 6/32 2024-04-20T02:44:00.681 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-20T02:44:00.682 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 27/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2024-04-20T02:44:00.683 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2024-04-20T02:44:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2024-04-20T02:44:13.796 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:44:14.162 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2024-04-20T02:44:14.162 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:44:14.335 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:44:14.335 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:44:14.335 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:14.335 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:14.335 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:14.335 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:44:14.335 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:14.335 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:14.335 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend-2.6-1.el8.noarch 2024-04-20T02:44:14.336 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:44:14.337 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:44:14.520 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-base 2024-04-20T02:44:14.713 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-20T02:44:14.714 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-2.6-1.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-20T02:44:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-20T02:44:14.716 INFO:teuthology.orchestra.run.smithi105.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-20T02:44:14.716 INFO:teuthology.orchestra.run.smithi105.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-20T02:44:14.716 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-20T02:44:14.716 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-20T02:44:14.716 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-20T02:44:14.716 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-20T02:44:14.716 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:44:14.716 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:44:14.871 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:44:14.905 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-base 2024-04-20T02:44:15.255 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:44:16.935 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-20T02:44:17.266 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-20T02:44:17.267 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-20T02:44:17.581 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 kB/s | 4.5 kB 00:00 2024-04-20T02:44:19.444 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-base-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:44:19.632 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:44:19.633 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-04-20T02:44:19.633 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:44:19.760 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-base-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:44:19.766 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install cephadm 2024-04-20T02:44:19.947 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:44:19.948 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-20T02:44:19.949 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:44:20.099 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install cephadm 2024-04-20T02:44:20.111 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:44:20.445 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:44:22.257 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-20T02:44:22.582 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 40 kB/s | 4.1 kB 00:00 2024-04-20T02:44:22.591 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-20T02:44:22.972 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-20T02:44:24.981 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:44:24.981 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 173 k 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 173 k 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 692 k 2024-04-20T02:44:24.982 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:44:25.296 INFO:teuthology.orchestra.run.smithi138.stdout:cephadm-18.2.2-1252.g15eecde2.el8.noarch.rpm 550 kB/s | 173 kB 00:00 2024-04-20T02:44:25.297 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:44:25.297 INFO:teuthology.orchestra.run.smithi138.stdout:Total 548 kB/s | 173 kB 00:00 2024-04-20T02:44:25.297 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:44:25.305 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:44:25.305 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:44:25.307 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:44:25.316 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:44:25.325 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:44:25.326 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:44:25.326 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-20T02:44:25.326 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:44:25.327 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:44:25.327 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 173 k 2024-04-20T02:44:25.327 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:44:25.327 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:44:25.327 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:44:25.327 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-20T02:44:25.327 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:44:25.328 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 173 k 2024-04-20T02:44:25.328 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 692 k 2024-04-20T02:44:25.328 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:44:25.369 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:44:25.646 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm-18.2.2-1252.g15eecde2.el8.noarch.rpm 547 kB/s | 173 kB 00:00 2024-04-20T02:44:25.646 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:44:25.647 INFO:teuthology.orchestra.run.smithi105.stdout:Total 545 kB/s | 173 kB 00:00 2024-04-20T02:44:25.647 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:44:25.652 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:44:25.652 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:44:25.654 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:44:25.663 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:44:25.716 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:44:27.243 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 1/1 2024-04-20T02:44:27.429 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 1/1 2024-04-20T02:44:27.629 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 1/1 2024-04-20T02:44:27.746 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 1/1 2024-04-20T02:44:27.928 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 1/1 2024-04-20T02:44:28.129 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 1/1 2024-04-20T02:44:36.468 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 1/1 2024-04-20T02:44:36.468 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:44:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:44:37.022 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:44:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:44:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:44:37.022 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:44:37.216 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-immutable-object-cache 2024-04-20T02:44:37.579 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:44:39.467 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-20T02:44:39.836 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-20T02:44:40.858 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 1/1 2024-04-20T02:44:40.858 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:44:41.402 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:44:41.402 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:44:41.402 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:44:41.402 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:44:41.402 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:44:41.559 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-immutable-object-cache 2024-04-20T02:44:41.920 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:44:42.193 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:44:42.193 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:44:42.194 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-04-20T02:44:42.194 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:44:42.194 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:44:42.194 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 166 k 2024-04-20T02:44:42.194 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:44:42.194 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:44:42.194 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:44:42.194 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-04-20T02:44:42.194 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:44:42.195 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 166 k 2024-04-20T02:44:42.195 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 503 k 2024-04-20T02:44:42.195 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:44:42.416 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-immutable-object-cache-18.2.2-1252.g15eecd 755 kB/s | 166 kB 00:00 2024-04-20T02:44:42.416 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:44:42.416 INFO:teuthology.orchestra.run.smithi138.stdout:Total 751 kB/s | 166 kB 00:00 2024-04-20T02:44:42.416 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:44:42.426 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:44:42.427 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:44:42.480 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:44:42.488 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:44:42.611 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:44:42.792 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1252.g15eecde2. 1/1 2024-04-20T02:44:43.184 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1252.g15eecde2. 1/1 2024-04-20T02:44:43.783 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-20T02:44:44.119 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-04-20T02:44:46.471 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 166 k 2024-04-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-20T02:44:46.472 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:44:46.473 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 166 k 2024-04-20T02:44:46.473 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 503 k 2024-04-20T02:44:46.473 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:44:46.663 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-immutable-object-cache-18.2.2-1252.g15eecd 869 kB/s | 166 kB 00:00 2024-04-20T02:44:46.664 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:44:46.664 INFO:teuthology.orchestra.run.smithi105.stdout:Total 863 kB/s | 166 kB 00:00 2024-04-20T02:44:46.664 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:44:46.675 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:44:46.676 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:44:46.729 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:44:46.737 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:44:46.859 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:44:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1252.g15eecde2. 1/1 2024-04-20T02:44:47.452 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1252.g15eecde2. 1/1 2024-04-20T02:44:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1252.g15eecde2. 1/1 2024-04-20T02:44:51.107 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:44:51.657 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:44:51.658 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:44:51.658 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:51.658 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:44:51.658 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:44:51.857 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr 2024-04-20T02:44:52.207 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:44:54.131 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-20T02:44:54.497 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-20T02:44:56.790 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-mgr-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:44:56.982 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:44:56.982 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-04-20T02:44:56.982 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:44:57.122 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-dashboard 2024-04-20T02:44:57.471 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:44:58.288 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1252.g15eecde2. 1/1 2024-04-20T02:44:58.288 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:44:58.841 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:44:58.841 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:44:58.842 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:44:58.842 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:44:58.842 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:44:59.131 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr 2024-04-20T02:44:59.324 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-20T02:44:59.480 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:44:59.691 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-20T02:45:01.107 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 39 kB/s | 4.1 kB 00:00 2024-04-20T02:45:01.457 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-20T02:45:02.075 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:45:02.076 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:45:02.076 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-20T02:45:02.076 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 3.5 M 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 24 k 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 15 k 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout:Install 5 Packages 2024-04-20T02:45:02.077 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:45:02.078 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 3.8 M 2024-04-20T02:45:02.078 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 87 M 2024-04-20T02:45:02.078 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:45:02.428 INFO:teuthology.orchestra.run.smithi138.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1252.g15e 113 kB/s | 24 kB 00:00 2024-04-20T02:45:02.453 INFO:teuthology.orchestra.run.smithi138.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1252.g15ee 63 kB/s | 15 kB 00:00 2024-04-20T02:45:02.628 INFO:teuthology.orchestra.run.smithi138.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 168 kB/s | 34 kB 00:00 2024-04-20T02:45:02.720 INFO:teuthology.orchestra.run.smithi138.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 736 kB/s | 196 kB 00:00 2024-04-20T02:45:03.740 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-mgr-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:45:03.928 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:45:03.929 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-20T02:45:03.929 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:45:04.068 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-dashboard 2024-04-20T02:45:04.312 INFO:teuthology.orchestra.run.smithi138.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1252.g15eecde2 1.7 MB/s | 3.5 MB 00:02 2024-04-20T02:45:04.314 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:45:04.314 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.7 MB/s | 3.8 MB 00:02 2024-04-20T02:45:04.314 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:45:04.323 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:45:04.323 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:45:04.395 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:45:04.403 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:45:04.417 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:45:04.669 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:45:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-20T02:45:05.116 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-20T02:45:05.287 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1252.g15eecde2.el8.n 3/5 2024-04-20T02:45:06.416 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1252.g15eecde2.el8. 4/5 2024-04-20T02:45:06.568 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-04-20T02:45:06.727 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1252.g15eecde2.el8.noarc 5/5 2024-04-20T02:45:06.879 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 kB/s | 4.5 kB 00:00 2024-04-20T02:45:06.992 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1252.g15eecde2.el8.noarc 5/5 2024-04-20T02:45:06.993 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1252.g15eecde2.el8. 1/5 2024-04-20T02:45:06.993 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1252.g15eecde2.el8.noarc 2/5 2024-04-20T02:45:06.993 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1252.g15eecde2.el8.n 3/5 2024-04-20T02:45:06.993 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-20T02:45:09.366 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 3.5 M 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 24 k 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 15 k 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-20T02:45:09.368 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:45:09.369 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:45:09.369 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:45:09.369 INFO:teuthology.orchestra.run.smithi105.stdout:Install 5 Packages 2024-04-20T02:45:09.369 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:45:09.369 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 3.8 M 2024-04-20T02:45:09.369 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 87 M 2024-04-20T02:45:09.369 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:45:09.602 INFO:teuthology.orchestra.run.smithi105.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1252.g15ee 100 kB/s | 15 kB 00:00 2024-04-20T02:45:09.702 INFO:teuthology.orchestra.run.smithi105.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1252.g15e 96 kB/s | 24 kB 00:00 2024-04-20T02:45:09.727 INFO:teuthology.orchestra.run.smithi105.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 269 kB/s | 34 kB 00:00 2024-04-20T02:45:09.827 INFO:teuthology.orchestra.run.smithi105.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 1.5 MB/s | 196 kB 00:00 2024-04-20T02:45:10.319 INFO:teuthology.orchestra.run.smithi105.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1252.g15eecde2 4.1 MB/s | 3.5 MB 00:00 2024-04-20T02:45:10.321 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:45:10.321 INFO:teuthology.orchestra.run.smithi105.stdout:Total 4.0 MB/s | 3.8 MB 00:00 2024-04-20T02:45:10.322 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:45:10.330 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:45:10.331 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:45:10.397 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:45:10.406 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:45:10.666 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:45:10.921 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-20T02:45:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-20T02:45:11.307 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1252.g15eecde2.el8.n 3/5 2024-04-20T02:45:12.445 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1252.g15eecde2.el8. 4/5 2024-04-20T02:45:12.726 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1252.g15eecde2.el8.noarc 5/5 2024-04-20T02:45:12.989 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1252.g15eecde2.el8.noarc 5/5 2024-04-20T02:45:12.990 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1252.g15eecde2.el8. 1/5 2024-04-20T02:45:12.990 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1252.g15eecde2.el8.noarc 2/5 2024-04-20T02:45:12.990 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1252.g15eecde2.el8.n 3/5 2024-04-20T02:45:12.990 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-20T02:45:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-20T02:45:20.347 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:45:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:45:20.931 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:45:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:45:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:45:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:45:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-20T02:45:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-20T02:45:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:45:20.932 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:45:21.123 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-20T02:45:21.481 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:45:23.578 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-20T02:45:23.868 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 55 kB/s | 4.5 kB 00:00 2024-04-20T02:45:25.785 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-20T02:45:25.785 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:45:26.292 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:45:26.293 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================================= 2024-04-20T02:45:26.294 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-20T02:45:26.294 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================================= 2024-04-20T02:45:26.294 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:45:26.294 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 7.4 M 2024-04-20T02:45:26.294 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-04-20T02:45:26.294 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-20T02:45:26.294 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-20T02:45:26.294 INFO:teuthology.orchestra.run.smithi138.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-20T02:45:26.294 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-20T02:45:26.294 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-20T02:45:26.295 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-20T02:45:26.295 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-20T02:45:26.295 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:45:26.295 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:45:26.295 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================================= 2024-04-20T02:45:26.295 INFO:teuthology.orchestra.run.smithi138.stdout:Install 8 Packages 2024-04-20T02:45:26.295 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:45:26.296 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 36 M 2024-04-20T02:45:26.296 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 207 M 2024-04-20T02:45:26.296 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:45:26.345 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:45:26.345 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:45:26.345 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:45:26.345 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:45:26.345 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:45:26.345 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-20T02:45:26.345 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-20T02:45:26.345 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:45:26.346 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:45:26.444 INFO:teuthology.orchestra.run.smithi138.stdout:(1/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-04-20T02:45:26.478 INFO:teuthology.orchestra.run.smithi138.stdout:(2/8): libgfortran-8.5.0-20.el8.x86_64.rpm 3.5 MB/s | 645 kB 00:00 2024-04-20T02:45:26.540 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-20T02:45:26.570 INFO:teuthology.orchestra.run.smithi138.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-04-20T02:45:26.678 INFO:teuthology.orchestra.run.smithi138.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 16 MB/s | 3.7 MB 00:00 2024-04-20T02:45:26.804 INFO:teuthology.orchestra.run.smithi138.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 22 MB/s | 5.2 MB 00:00 2024-04-20T02:45:26.895 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:45:26.946 INFO:teuthology.orchestra.run.smithi138.stdout:(6/8): openblas-0.3.15-6.el8.x86_64.rpm 19 MB/s | 5.0 MB 00:00 2024-04-20T02:45:27.204 INFO:teuthology.orchestra.run.smithi138.stdout:(7/8): ceph-mgr-diskprediction-local-18.2.2-125 8.1 MB/s | 7.4 MB 00:00 2024-04-20T02:45:27.417 INFO:teuthology.orchestra.run.smithi138.stdout:(8/8): python3-scipy-1.0.0-21.module+el8.9.0+20 22 MB/s | 14 MB 00:00 2024-04-20T02:45:27.418 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:45:27.418 INFO:teuthology.orchestra.run.smithi138.stdout:Total 32 MB/s | 36 MB 00:01 2024-04-20T02:45:27.662 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:45:27.681 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:45:27.681 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:45:27.993 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:45:28.002 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:45:28.592 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:45:28.615 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2024-04-20T02:45:28.801 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-20T02:45:28.926 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-20T02:45:28.937 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 4.5 kB 00:00 2024-04-20T02:45:29.110 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-20T02:45:29.738 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-20T02:45:29.934 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-20T02:45:30.552 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-20T02:45:30.768 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-20T02:45:31.284 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:45:31.285 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================================= 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================================= 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 7.4 M 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-20T02:45:31.286 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-20T02:45:31.287 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:45:31.287 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:45:31.287 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================================= 2024-04-20T02:45:31.287 INFO:teuthology.orchestra.run.smithi105.stdout:Install 8 Packages 2024-04-20T02:45:31.287 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:45:31.287 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 36 M 2024-04-20T02:45:31.287 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 207 M 2024-04-20T02:45:31.287 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:45:31.427 INFO:teuthology.orchestra.run.smithi105.stdout:(1/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.2 MB/s | 172 kB 00:00 2024-04-20T02:45:31.476 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-20T02:45:31.535 INFO:teuthology.orchestra.run.smithi105.stdout:(2/8): libgfortran-8.5.0-20.el8.x86_64.rpm 2.5 MB/s | 645 kB 00:00 2024-04-20T02:45:31.627 INFO:teuthology.orchestra.run.smithi105.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2024-04-20T02:45:31.653 INFO:teuthology.orchestra.run.smithi105.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2024-04-20T02:45:31.838 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T02:45:31.861 INFO:teuthology.orchestra.run.smithi105.stdout:(5/8): openblas-0.3.15-6.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2024-04-20T02:45:31.962 INFO:teuthology.orchestra.run.smithi105.stdout:(6/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 16 MB/s | 5.2 MB 00:00 2024-04-20T02:45:32.068 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T02:45:32.254 INFO:teuthology.orchestra.run.smithi105.stdout:(7/8): ceph-mgr-diskprediction-local-18.2.2-125 7.7 MB/s | 7.4 MB 00:00 2024-04-20T02:45:32.454 INFO:teuthology.orchestra.run.smithi105.stdout:(8/8): python3-scipy-1.0.0-21.module+el8.9.0+20 23 MB/s | 14 MB 00:00 2024-04-20T02:45:32.455 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:45:32.455 INFO:teuthology.orchestra.run.smithi105.stdout:Total 31 MB/s | 36 MB 00:01 2024-04-20T02:45:32.701 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:45:32.719 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:45:32.719 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:45:33.028 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:45:33.036 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:45:33.629 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:45:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-20T02:45:33.849 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T02:45:33.947 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-20T02:45:34.154 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-20T02:45:34.778 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-20T02:45:34.986 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-20T02:45:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-20T02:45:35.350 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1252.g15eecde 8/8 2024-04-20T02:45:35.601 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-20T02:45:35.658 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1252.g15eecde 8/8 2024-04-20T02:45:35.658 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1252.g15eecde 1/8 2024-04-20T02:45:35.658 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-20T02:45:35.658 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2024-04-20T02:45:35.658 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-20T02:45:35.659 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T02:45:35.659 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2024-04-20T02:45:35.659 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2024-04-20T02:45:35.837 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-20T02:45:36.556 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-20T02:45:36.966 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T02:45:37.187 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T02:45:38.944 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-20T02:45:40.230 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-20T02:45:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1252.g15eecde 8/8 2024-04-20T02:45:40.750 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1252.g15eecde 8/8 2024-04-20T02:45:40.751 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1252.g15eecde 1/8 2024-04-20T02:45:40.751 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-20T02:45:40.751 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2024-04-20T02:45:40.751 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-20T02:45:40.751 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-20T02:45:40.751 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2024-04-20T02:45:40.751 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2024-04-20T02:45:48.891 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2024-04-20T02:45:48.892 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:45:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:45:49.447 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:45:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:45:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-20T02:45:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-20T02:45:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-20T02:45:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-20T02:45:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-20T02:45:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-20T02:45:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-20T02:45:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:45:49.448 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:45:49.652 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-rook 2024-04-20T02:45:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2024-04-20T02:45:49.740 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:45:50.005 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:45:50.329 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:45:50.329 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:45:50.329 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:45:50.329 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-20T02:45:50.330 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-20T02:45:50.330 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-20T02:45:50.330 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-20T02:45:50.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-20T02:45:50.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-20T02:45:50.330 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-20T02:45:50.330 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:45:50.330 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:45:50.489 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-rook 2024-04-20T02:45:50.839 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:45:51.756 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 kB/s | 4.1 kB 00:00 2024-04-20T02:45:52.058 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 kB/s | 4.5 kB 00:00 2024-04-20T02:45:52.777 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-20T02:45:53.143 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-20T02:45:54.425 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:45:54.427 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================== 2024-04-20T02:45:54.427 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-20T02:45:54.427 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================== 2024-04-20T02:45:54.427 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:45:54.427 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 49 k 2024-04-20T02:45:54.427 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-04-20T02:45:54.427 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-20T02:45:54.427 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-20T02:45:54.427 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-20T02:45:54.427 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================== 2024-04-20T02:45:54.428 INFO:teuthology.orchestra.run.smithi138.stdout:Install 14 Packages 2024-04-20T02:45:54.429 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:45:54.429 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 2.0 M 2024-04-20T02:45:54.429 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 25 M 2024-04-20T02:45:54.429 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:45:54.890 INFO:teuthology.orchestra.run.smithi138.stdout:(1/14): python3-cachetools-3.1.1-4.el8.noarch.r 270 kB/s | 33 kB 00:00 2024-04-20T02:45:54.916 INFO:teuthology.orchestra.run.smithi138.stdout:(2/14): python3-certifi-2018.10.15-7.el8.noarch 110 kB/s | 16 kB 00:00 2024-04-20T02:45:54.941 INFO:teuthology.orchestra.run.smithi138.stdout:(3/14): ceph-mgr-rook-18.2.2-1252.g15eecde2.el8 284 kB/s | 49 kB 00:00 2024-04-20T02:45:54.966 INFO:teuthology.orchestra.run.smithi138.stdout:(4/14): python3-google-auth-1.1.1-10.el8.noarch 1.0 MB/s | 81 kB 00:00 2024-04-20T02:45:55.040 INFO:teuthology.orchestra.run.smithi138.stdout:(5/14): python3-websocket-client-0.56.0-5.el8.n 888 kB/s | 61 kB 00:00 2024-04-20T02:45:55.092 INFO:teuthology.orchestra.run.smithi138.stdout:(6/14): python3-kubernetes-11.0.0-6.el8.noarch. 6.6 MB/s | 1.2 MB 00:00 2024-04-20T02:45:55.126 INFO:teuthology.orchestra.run.smithi138.stdout:(7/14): python3-rsa-4.9-2.el8.noarch.rpm 348 kB/s | 64 kB 00:00 2024-04-20T02:45:55.218 INFO:teuthology.orchestra.run.smithi138.stdout:(8/14): python3-jwt-1.6.1-2.el8.noarch.rpm 241 kB/s | 43 kB 00:00 2024-04-20T02:45:55.251 INFO:teuthology.orchestra.run.smithi138.stdout:(9/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.0 MB/s | 155 kB 00:00 2024-04-20T02:45:55.276 INFO:teuthology.orchestra.run.smithi138.stdout:(10/14): python3-requests-oauthlib-1.0.0-1.el8. 294 kB/s | 43 kB 00:00 2024-04-20T02:45:55.368 INFO:teuthology.orchestra.run.smithi138.stdout:(11/14): python3-jsonpointer-1.10-11.el8.noarch 133 kB/s | 20 kB 00:00 2024-04-20T02:45:55.419 INFO:teuthology.orchestra.run.smithi138.stdout:(12/14): python3-jsonpatch-1.21-2.el8.noarch.rp 187 kB/s | 27 kB 00:00 2024-04-20T02:45:55.452 INFO:teuthology.orchestra.run.smithi138.stdout:(13/14): python3-pyasn1-modules-0.3.7-6.el8.noa 548 kB/s | 110 kB 00:00 2024-04-20T02:45:55.494 INFO:teuthology.orchestra.run.smithi138.stdout:(14/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2024-04-20T02:45:55.497 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:45:55.497 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-04-20T02:45:55.524 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:45:55.525 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================== 2024-04-20T02:45:55.525 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================== 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 49 k 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-20T02:45:55.526 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-20T02:45:55.527 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-20T02:45:55.527 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-20T02:45:55.527 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-20T02:45:55.527 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-20T02:45:55.527 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-20T02:45:55.527 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:45:55.527 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:45:55.527 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================== 2024-04-20T02:45:55.527 INFO:teuthology.orchestra.run.smithi105.stdout:Install 14 Packages 2024-04-20T02:45:55.527 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:45:55.528 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.0 M 2024-04-20T02:45:55.528 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 25 M 2024-04-20T02:45:55.528 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:45:55.569 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:45:55.584 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:45:55.584 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:45:55.665 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:45:55.673 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:45:55.773 INFO:teuthology.orchestra.run.smithi105.stdout:(1/14): python3-certifi-2018.10.15-7.el8.noarch 138 kB/s | 16 kB 00:00 2024-04-20T02:45:55.798 INFO:teuthology.orchestra.run.smithi105.stdout:(2/14): python3-cachetools-3.1.1-4.el8.noarch.r 230 kB/s | 33 kB 00:00 2024-04-20T02:45:55.823 INFO:teuthology.orchestra.run.smithi105.stdout:(3/14): ceph-mgr-rook-18.2.2-1252.g15eecde2.el8 291 kB/s | 49 kB 00:00 2024-04-20T02:45:55.865 INFO:teuthology.orchestra.run.smithi105.stdout:(4/14): python3-google-auth-1.1.1-10.el8.noarch 883 kB/s | 81 kB 00:00 2024-04-20T02:45:55.987 INFO:teuthology.orchestra.run.smithi105.stdout:(5/14): python3-websocket-client-0.56.0-5.el8.n 504 kB/s | 61 kB 00:00 2024-04-20T02:45:56.016 INFO:teuthology.orchestra.run.smithi105.stdout:(6/14): python3-rsa-4.9-2.el8.noarch.rpm 334 kB/s | 64 kB 00:00 2024-04-20T02:45:56.050 INFO:teuthology.orchestra.run.smithi105.stdout:(7/14): python3-kubernetes-11.0.0-6.el8.noarch. 4.6 MB/s | 1.2 MB 00:00 2024-04-20T02:45:56.217 INFO:teuthology.orchestra.run.smithi105.stdout:(8/14): python3-jwt-1.6.1-2.el8.noarch.rpm 185 kB/s | 43 kB 00:00 2024-04-20T02:45:56.246 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:45:56.250 INFO:teuthology.orchestra.run.smithi105.stdout:(9/14): python3-requests-oauthlib-1.0.0-1.el8.n 225 kB/s | 43 kB 00:00 2024-04-20T02:45:56.275 INFO:teuthology.orchestra.run.smithi105.stdout:(10/14): python3-oauthlib-2.1.0-1.el8.noarch.rp 598 kB/s | 155 kB 00:00 2024-04-20T02:45:56.334 INFO:teuthology.orchestra.run.smithi105.stdout:(11/14): python3-jsonpointer-1.10-11.el8.noarch 171 kB/s | 20 kB 00:00 2024-04-20T02:45:56.376 INFO:teuthology.orchestra.run.smithi105.stdout:(12/14): python3-pyasn1-modules-0.3.7-6.el8.noa 877 kB/s | 110 kB 00:00 2024-04-20T02:45:56.459 INFO:teuthology.orchestra.run.smithi105.stdout:(13/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2024-04-20T02:45:56.485 INFO:teuthology.orchestra.run.smithi105.stdout:(14/14): python3-jsonpatch-1.21-2.el8.noarch.rp 127 kB/s | 27 kB 00:00 2024-04-20T02:45:56.485 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:45:56.485 INFO:teuthology.orchestra.run.smithi105.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-04-20T02:45:56.523 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-20T02:45:56.556 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:45:56.571 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:45:56.571 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:45:56.652 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:45:56.661 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:45:56.775 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-20T02:45:56.995 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-20T02:45:57.146 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/14 2024-04-20T02:45:57.194 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:45:57.350 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/14 2024-04-20T02:45:57.434 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-20T02:45:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-04-20T02:45:57.678 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-20T02:45:57.891 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-20T02:45:57.945 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-04-20T02:45:58.071 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/14 2024-04-20T02:45:58.226 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-04-20T02:45:58.244 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/14 2024-04-20T02:45:58.471 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-20T02:45:58.514 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-04-20T02:45:58.704 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-20T02:45:58.797 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-04-20T02:45:58.962 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-20T02:45:59.053 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-04-20T02:45:59.282 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-20T02:45:59.514 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-20T02:45:59.788 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-20T02:46:00.054 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-20T02:46:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-20T02:46:00.546 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-rook-2:18.2.2-1252.g15eecde2.el8.noarch 14/14 2024-04-20T02:46:00.819 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1252.g15eecde2.el8.noarch 14/14 2024-04-20T02:46:00.819 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1252.g15eecde2.el8.noarch 1/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/14 2024-04-20T02:46:00.820 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-04-20T02:46:00.845 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-20T02:46:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-20T02:46:01.400 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:18.2.2-1252.g15eecde2.el8.noarch 14/14 2024-04-20T02:46:01.682 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1252.g15eecde2.el8.noarch 14/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1252.g15eecde2.el8.noarch 1/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/14 2024-04-20T02:46:01.683 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-04-20T02:46:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 14/14 2024-04-20T02:46:10.493 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:46:11.063 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:46:11.064 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:46:11.064 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:46:11.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-20T02:46:11.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-20T02:46:11.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-20T02:46:11.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-20T02:46:11.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-20T02:46:11.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-20T02:46:11.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-20T02:46:11.064 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-20T02:46:11.065 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-20T02:46:11.065 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-20T02:46:11.065 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-20T02:46:11.065 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-20T02:46:11.065 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-20T02:46:11.065 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:46:11.065 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:46:11.264 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-cephadm 2024-04-20T02:46:11.624 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:46:13.508 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 57 kB/s | 4.1 kB 00:00 2024-04-20T02:46:13.834 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2024-04-20T02:46:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 14/14 2024-04-20T02:46:14.225 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:46:14.831 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:14.831 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:46:14.831 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:46:14.831 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-20T02:46:14.831 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-20T02:46:14.831 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-20T02:46:14.831 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:14.832 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:46:15.044 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-cephadm 2024-04-20T02:46:15.401 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:46:16.241 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:46:16.241 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================================== 2024-04-20T02:46:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================================== 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 138 k 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================================== 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout:Install 5 Packages 2024-04-20T02:46:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:46:16.243 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 6.0 M 2024-04-20T02:46:16.243 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 26 M 2024-04-20T02:46:16.243 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:46:16.862 INFO:teuthology.orchestra.run.smithi138.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 216 kB/s | 60 kB 00:00 2024-04-20T02:46:16.888 INFO:teuthology.orchestra.run.smithi138.stdout:(2/5): ceph-mgr-cephadm-18.2.2-1252.g15eecde2.e 453 kB/s | 138 kB 00:00 2024-04-20T02:46:17.021 INFO:teuthology.orchestra.run.smithi138.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.1 MB/s | 476 kB 00:00 2024-04-20T02:46:17.055 INFO:teuthology.orchestra.run.smithi138.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.1 MB/s | 538 kB 00:00 2024-04-20T02:46:17.199 INFO:teuthology.orchestra.run.smithi138.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2024-04-20T02:46:17.201 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:46:17.201 INFO:teuthology.orchestra.run.smithi138.stdout:Total 6.2 MB/s | 6.0 MB 00:00 2024-04-20T02:46:17.255 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:46:17.260 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-20T02:46:17.267 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:46:17.267 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:46:17.357 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:46:17.366 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:46:17.611 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-20T02:46:18.362 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:46:18.732 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-20T02:46:19.020 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-20T02:46:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-20T02:46:19.701 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-20T02:46:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 5/5 2024-04-20T02:46:20.003 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:46:20.003 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================================== 2024-04-20T02:46:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-20T02:46:20.003 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================================== 2024-04-20T02:46:20.003 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:46:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 138 k 2024-04-20T02:46:20.004 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-20T02:46:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-20T02:46:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-20T02:46:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-20T02:46:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-20T02:46:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:20.004 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:46:20.004 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================================== 2024-04-20T02:46:20.004 INFO:teuthology.orchestra.run.smithi105.stdout:Install 5 Packages 2024-04-20T02:46:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:20.005 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 6.0 M 2024-04-20T02:46:20.005 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 26 M 2024-04-20T02:46:20.005 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:46:20.181 INFO:teuthology.orchestra.run.smithi105.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 526 kB/s | 60 kB 00:00 2024-04-20T02:46:20.208 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 5/5 2024-04-20T02:46:20.208 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 1/5 2024-04-20T02:46:20.208 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-20T02:46:20.208 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-20T02:46:20.208 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-20T02:46:20.281 INFO:teuthology.orchestra.run.smithi105.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.2 MB/s | 476 kB 00:00 2024-04-20T02:46:20.454 INFO:teuthology.orchestra.run.smithi105.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2024-04-20T02:46:20.482 INFO:teuthology.orchestra.run.smithi105.stdout:(4/5): ceph-mgr-cephadm-18.2.2-1252.g15eecde2.e 332 kB/s | 138 kB 00:00 2024-04-20T02:46:20.507 INFO:teuthology.orchestra.run.smithi105.stdout:(5/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.3 MB/s | 538 kB 00:00 2024-04-20T02:46:20.508 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:46:20.508 INFO:teuthology.orchestra.run.smithi105.stdout:Total 12 MB/s | 6.0 MB 00:00 2024-04-20T02:46:20.561 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:46:20.572 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:46:20.572 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:46:20.660 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:46:20.668 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:46:21.631 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:46:22.009 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-20T02:46:22.332 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-20T02:46:22.694 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-20T02:46:22.980 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-20T02:46:23.414 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 5/5 2024-04-20T02:46:23.706 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 5/5 2024-04-20T02:46:23.706 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 1/5 2024-04-20T02:46:23.706 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-20T02:46:23.706 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-20T02:46:23.706 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-20T02:46:34.006 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-20T02:46:34.006 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:46:34.571 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:46:34.571 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:46:34.571 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:46:34.571 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-20T02:46:34.571 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-20T02:46:34.571 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-20T02:46:34.571 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-20T02:46:34.572 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:46:34.572 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:46:34.730 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-fuse 2024-04-20T02:46:35.087 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:46:35.487 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-20T02:46:35.487 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:46:36.065 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:36.065 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:46:36.065 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:46:36.065 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-20T02:46:36.066 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-20T02:46:36.066 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-20T02:46:36.066 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-20T02:46:36.066 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:36.066 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:46:36.228 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-fuse 2024-04-20T02:46:36.589 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:46:37.190 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-20T02:46:37.546 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-20T02:46:38.571 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 kB/s | 4.1 kB 00:00 2024-04-20T02:46:38.919 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-04-20T02:46:39.930 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:46:39.931 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:46:39.931 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-20T02:46:39.931 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:46:39.931 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:46:39.931 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse 2024-04-20T02:46:39.931 INFO:teuthology.orchestra.run.smithi138.stdout: x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 859 k 2024-04-20T02:46:39.931 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-04-20T02:46:39.932 INFO:teuthology.orchestra.run.smithi138.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-20T02:46:39.932 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:46:39.932 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:46:39.932 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:46:39.932 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2024-04-20T02:46:39.932 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:46:39.932 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 942 k 2024-04-20T02:46:39.933 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 3.0 M 2024-04-20T02:46:39.933 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:46:40.083 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 553 kB/s | 83 kB 00:00 2024-04-20T02:46:40.217 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-fuse-18.2.2-1252.g15eecde2.el8.x86_ 3.0 MB/s | 859 kB 00:00 2024-04-20T02:46:40.217 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:46:40.217 INFO:teuthology.orchestra.run.smithi138.stdout:Total 3.2 MB/s | 942 kB 00:00 2024-04-20T02:46:40.229 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:46:40.250 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:46:40.250 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:46:40.335 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:46:40.343 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:46:40.551 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:46:40.847 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-20T02:46:41.286 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 2/2 2024-04-20T02:46:41.397 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:46:41.398 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:46:41.398 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-20T02:46:41.398 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:46:41.398 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:46:41.398 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse 2024-04-20T02:46:41.398 INFO:teuthology.orchestra.run.smithi105.stdout: x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 859 k 2024-04-20T02:46:41.398 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-20T02:46:41.398 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-20T02:46:41.399 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:41.399 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:46:41.399 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:46:41.399 INFO:teuthology.orchestra.run.smithi105.stdout:Install 2 Packages 2024-04-20T02:46:41.399 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:41.399 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 942 k 2024-04-20T02:46:41.399 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 3.0 M 2024-04-20T02:46:41.399 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:46:41.560 INFO:teuthology.orchestra.run.smithi105.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 517 kB/s | 83 kB 00:00 2024-04-20T02:46:41.661 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 2/2 2024-04-20T02:46:41.661 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 1/2 2024-04-20T02:46:42.018 INFO:teuthology.orchestra.run.smithi105.stdout:(2/2): ceph-fuse-18.2.2-1252.g15eecde2.el8.x86_ 1.4 MB/s | 859 kB 00:00 2024-04-20T02:46:42.019 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:46:42.019 INFO:teuthology.orchestra.run.smithi105.stdout:Total 1.5 MB/s | 942 kB 00:00 2024-04-20T02:46:42.031 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:46:42.053 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:46:42.053 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:46:42.139 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:46:42.148 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:46:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:46:42.619 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-20T02:46:42.853 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 2/2 2024-04-20T02:46:43.231 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 2/2 2024-04-20T02:46:43.231 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 1/2 2024-04-20T02:46:53.424 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-20T02:46:53.424 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:46:54.028 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:54.029 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:46:54.029 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-20T02:46:54.029 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:54.029 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:46:54.194 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-volume 2024-04-20T02:46:54.557 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:46:55.126 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-20T02:46:55.127 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:46:55.684 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:46:55.685 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:46:55.685 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-20T02:46:55.685 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:46:55.685 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:46:55.851 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-volume 2024-04-20T02:46:56.211 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:46:56.284 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-20T02:46:56.542 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 107 kB/s | 4.5 kB 00:00 2024-04-20T02:46:58.090 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-04-20T02:46:58.432 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-04-20T02:46:58.935 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:46:58.936 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================== 2024-04-20T02:46:58.936 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-04-20T02:46:58.936 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================== 2024-04-20T02:46:58.936 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:46:58.936 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 263 k 2024-04-20T02:46:58.936 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-04-20T02:46:58.936 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-20T02:46:58.936 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:58.937 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:46:58.937 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================================== 2024-04-20T02:46:58.937 INFO:teuthology.orchestra.run.smithi105.stdout:Install 2 Packages 2024-04-20T02:46:58.937 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:46:58.937 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 311 k 2024-04-20T02:46:58.937 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 1.4 M 2024-04-20T02:46:58.937 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:46:59.116 INFO:teuthology.orchestra.run.smithi105.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 270 kB/s | 48 kB 00:00 2024-04-20T02:46:59.733 INFO:teuthology.orchestra.run.smithi105.stdout:(2/2): ceph-volume-18.2.2-1252.g15eecde2.el8.no 330 kB/s | 263 kB 00:00 2024-04-20T02:46:59.734 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:46:59.734 INFO:teuthology.orchestra.run.smithi105.stdout:Total 390 kB/s | 311 kB 00:00 2024-04-20T02:46:59.747 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:46:59.760 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:46:59.761 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:46:59.800 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:46:59.809 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:46:59.974 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:47:00.349 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-20T02:47:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-volume-2:18.2.2-1252.g15eecde2.el8.noarch 2/2 2024-04-20T02:47:00.909 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:47:00.910 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================== 2024-04-20T02:47:00.910 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-04-20T02:47:00.910 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================== 2024-04-20T02:47:00.910 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:47:00.910 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume noarch 2:18.2.2-1252.g15eecde2.el8 ceph-noarch 263 k 2024-04-20T02:47:00.910 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-04-20T02:47:00.910 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-20T02:47:00.910 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:00.910 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:47:00.910 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================================== 2024-04-20T02:47:00.911 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2024-04-20T02:47:00.911 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:00.911 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 311 k 2024-04-20T02:47:00.911 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 1.4 M 2024-04-20T02:47:00.911 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:47:01.050 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-volume-2:18.2.2-1252.g15eecde2.el8.noarch 2/2 2024-04-20T02:47:01.051 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-volume-2:18.2.2-1252.g15eecde2.el8.noarch 1/2 2024-04-20T02:47:01.098 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 258 kB/s | 48 kB 00:00 2024-04-20T02:47:01.124 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-volume-18.2.2-1252.g15eecde2.el8.no 1.2 MB/s | 263 kB 00:00 2024-04-20T02:47:01.124 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:47:01.124 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.4 MB/s | 311 kB 00:00 2024-04-20T02:47:01.136 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:47:01.148 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:47:01.148 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:47:01.185 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:47:01.193 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:47:01.360 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:47:01.743 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-20T02:47:02.062 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-volume-2:18.2.2-1252.g15eecde2.el8.noarch 2/2 2024-04-20T02:47:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-volume-2:18.2.2-1252.g15eecde2.el8.noarch 2/2 2024-04-20T02:47:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-volume-2:18.2.2-1252.g15eecde2.el8.noarch 1/2 2024-04-20T02:47:13.441 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-20T02:47:13.441 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:47:14.045 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:47:14.045 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:47:14.045 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:47:14.045 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-20T02:47:14.045 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:47:14.045 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:47:14.122 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-20T02:47:14.122 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:47:14.189 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados-devel 2024-04-20T02:47:14.551 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:47:14.712 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:14.713 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:47:14.713 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume-2:18.2.2-1252.g15eecde2.el8.noarch 2024-04-20T02:47:14.713 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-20T02:47:14.713 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:14.713 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:47:14.853 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados-devel 2024-04-20T02:47:15.214 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:47:16.456 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2024-04-20T02:47:16.820 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-20T02:47:17.088 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2024-04-20T02:47:17.438 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-20T02:47:19.185 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:47:19.186 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:47:19.186 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-04-20T02:47:19.186 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:47:19.186 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:47:19.186 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 139 k 2024-04-20T02:47:19.186 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:47:19.186 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:47:19.187 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:47:19.187 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-20T02:47:19.187 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:47:19.187 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 139 k 2024-04-20T02:47:19.187 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 513 k 2024-04-20T02:47:19.187 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:47:19.382 INFO:teuthology.orchestra.run.smithi105.stdout:librados-devel-18.2.2-1252.g15eecde2.el8.x86_64 717 kB/s | 139 kB 00:00 2024-04-20T02:47:19.382 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:47:19.382 INFO:teuthology.orchestra.run.smithi105.stdout:Total 712 kB/s | 139 kB 00:00 2024-04-20T02:47:19.383 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:47:19.388 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:47:19.388 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:47:19.445 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:47:19.453 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:47:19.621 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:47:19.798 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:47:19.799 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:47:19.799 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-04-20T02:47:19.799 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:47:19.799 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:47:19.799 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 139 k 2024-04-20T02:47:19.799 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:19.799 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:47:19.800 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:47:19.800 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-04-20T02:47:19.800 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:19.800 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 139 k 2024-04-20T02:47:19.801 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 513 k 2024-04-20T02:47:19.801 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:47:19.856 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:20.090 INFO:teuthology.orchestra.run.smithi138.stdout:librados-devel-18.2.2-1252.g15eecde2.el8.x86_64 481 kB/s | 139 kB 00:00 2024-04-20T02:47:20.090 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:47:20.090 INFO:teuthology.orchestra.run.smithi138.stdout:Total 479 kB/s | 139 kB 00:00 2024-04-20T02:47:20.091 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:47:20.097 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:47:20.097 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:47:20.127 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:20.154 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:47:20.162 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:47:20.284 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:47:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librados-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:20.810 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:32.411 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:47:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:32.512 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:47:32.979 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:32.979 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:47:32.979 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:47:32.979 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:32.979 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:47:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:47:33.083 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:47:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:47:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:47:33.083 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:47:33.127 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs2 2024-04-20T02:47:33.236 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs2 2024-04-20T02:47:33.486 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:47:33.595 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:47:35.325 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-20T02:47:35.527 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-20T02:47:35.700 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-20T02:47:35.877 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-20T02:47:37.867 INFO:teuthology.orchestra.run.smithi138.stdout:Package libcephfs2-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:47:38.062 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:47:38.063 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-04-20T02:47:38.063 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:47:38.070 INFO:teuthology.orchestra.run.smithi105.stdout:Package libcephfs2-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:47:38.196 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs-devel 2024-04-20T02:47:38.263 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:47:38.264 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-20T02:47:38.264 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:47:38.395 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs-devel 2024-04-20T02:47:38.554 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:47:38.754 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:47:40.477 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-20T02:47:40.722 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 kB/s | 4.1 kB 00:00 2024-04-20T02:47:40.780 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 53 kB/s | 4.5 kB 00:00 2024-04-20T02:47:41.108 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-20T02:47:43.267 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:47:43.267 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:47:43.267 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-04-20T02:47:43.267 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:47:43.267 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:47:43.268 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 31 k 2024-04-20T02:47:43.268 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:43.268 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:47:43.268 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:47:43.268 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-04-20T02:47:43.268 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:43.268 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 31 k 2024-04-20T02:47:43.269 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 139 k 2024-04-20T02:47:43.269 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:47:43.411 INFO:teuthology.orchestra.run.smithi138.stdout:libcephfs-devel-18.2.2-1252.g15eecde2.el8.x86_6 216 kB/s | 31 kB 00:00 2024-04-20T02:47:43.412 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:47:43.412 INFO:teuthology.orchestra.run.smithi138.stdout:Total 215 kB/s | 31 kB 00:00 2024-04-20T02:47:43.412 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:47:43.416 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:47:43.416 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:47:43.428 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:47:43.437 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:47:43.495 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:47:43.515 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:47:43.516 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:47:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-04-20T02:47:43.516 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:47:43.516 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:47:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 31 k 2024-04-20T02:47:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:47:43.516 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:47:43.516 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:47:43.516 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-20T02:47:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:47:43.517 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 31 k 2024-04-20T02:47:43.517 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 139 k 2024-04-20T02:47:43.517 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:47:43.739 INFO:teuthology.orchestra.run.smithi105.stdout:libcephfs-devel-18.2.2-1252.g15eecde2.el8.x86_6 140 kB/s | 31 kB 00:00 2024-04-20T02:47:43.739 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:47:43.739 INFO:teuthology.orchestra.run.smithi105.stdout:Total 139 kB/s | 31 kB 00:00 2024-04-20T02:47:43.739 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:47:43.743 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:47:43.743 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:47:43.745 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:43.755 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:47:43.764 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:47:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:47:43.988 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:44.052 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:44.300 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:53.592 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:53.592 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:47:54.107 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:47:54.107 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:47:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:47:54.164 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:47:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:47:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:47:54.164 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:47:54.317 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados2 2024-04-20T02:47:54.669 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:54.669 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:47:54.669 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:47:54.669 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:47:54.669 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:47:54.679 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:47:54.813 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados2 2024-04-20T02:47:55.171 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:47:56.683 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-20T02:47:57.024 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-20T02:47:57.065 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-20T02:47:57.412 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-20T02:47:59.223 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:47:59.412 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:47:59.413 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-20T02:47:59.413 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:47:59.544 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librbd1 2024-04-20T02:47:59.575 INFO:teuthology.orchestra.run.smithi138.stdout:Package librados2-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:47:59.763 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:47:59.764 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-04-20T02:47:59.764 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:47:59.903 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:47:59.910 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librbd1 2024-04-20T02:48:00.267 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:01.861 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 47 kB/s | 4.1 kB 00:00 2024-04-20T02:48:02.035 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 47 kB/s | 4.1 kB 00:00 2024-04-20T02:48:02.186 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 4.5 kB 00:00 2024-04-20T02:48:02.359 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 4.5 kB 00:00 2024-04-20T02:48:04.358 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:48:04.550 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:48:04.551 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-20T02:48:04.551 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:48:04.649 INFO:teuthology.orchestra.run.smithi138.stdout:Package librbd1-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:48:04.675 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rados 2024-04-20T02:48:04.836 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:48:04.837 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-04-20T02:48:04.837 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:48:04.983 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rados 2024-04-20T02:48:05.033 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:05.340 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:06.971 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-20T02:48:07.170 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-04-20T02:48:07.342 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-20T02:48:07.527 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 47 kB/s | 4.5 kB 00:00 2024-04-20T02:48:09.522 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rados-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:48:09.703 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rados-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:48:09.715 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:48:09.717 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-20T02:48:09.717 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:48:09.861 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rgw 2024-04-20T02:48:09.891 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:48:09.892 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-04-20T02:48:09.892 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:48:10.043 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rgw 2024-04-20T02:48:10.221 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:10.401 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:11.954 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 kB/s | 4.1 kB 00:00 2024-04-20T02:48:12.212 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-20T02:48:12.246 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 kB/s | 4.5 kB 00:00 2024-04-20T02:48:12.502 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 66 kB/s | 4.5 kB 00:00 2024-04-20T02:48:14.533 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rgw-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:48:14.700 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rgw-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:48:14.719 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:48:14.720 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-20T02:48:14.720 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:48:14.860 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-cephfs 2024-04-20T02:48:14.891 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:48:14.893 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-04-20T02:48:14.893 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:48:15.031 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-cephfs 2024-04-20T02:48:15.216 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:15.389 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:17.159 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-20T02:48:17.297 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-20T02:48:17.527 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-20T02:48:17.645 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-04-20T02:48:19.811 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-cephfs-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:48:19.847 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-cephfs-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:48:20.004 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:48:20.005 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-20T02:48:20.005 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:48:20.035 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:48:20.036 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-04-20T02:48:20.036 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:48:20.144 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rbd 2024-04-20T02:48:20.183 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rbd 2024-04-20T02:48:20.502 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:20.539 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:22.299 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-20T02:48:22.478 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-20T02:48:22.692 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-20T02:48:22.778 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 53 kB/s | 4.5 kB 00:00 2024-04-20T02:48:24.870 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rbd-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:48:24.962 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rbd-2:18.2.2-1252.g15eecde2.el8.x86_64 is already installed. 2024-04-20T02:48:25.061 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:48:25.062 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-04-20T02:48:25.062 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:48:25.153 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:48:25.154 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-04-20T02:48:25.154 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:48:25.206 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-fuse 2024-04-20T02:48:25.296 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-fuse 2024-04-20T02:48:25.564 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:25.655 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:27.399 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-20T02:48:27.686 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-20T02:48:27.730 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-20T02:48:28.083 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-04-20T02:48:30.123 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 95 k 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 95 k 2024-04-20T02:48:30.124 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 280 k 2024-04-20T02:48:30.125 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:48:30.428 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-fuse-18.2.2-1252.g15eecde2.el8.x86_64.rpm 314 kB/s | 95 kB 00:00 2024-04-20T02:48:30.428 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:48:30.428 INFO:teuthology.orchestra.run.smithi105.stdout:Total 313 kB/s | 95 kB 00:00 2024-04-20T02:48:30.429 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:48:30.436 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:48:30.436 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:48:30.437 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-04-20T02:48:30.437 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:48:30.437 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:48:30.437 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 95 k 2024-04-20T02:48:30.437 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:48:30.437 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:48:30.437 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:48:30.437 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-04-20T02:48:30.437 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:48:30.438 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 95 k 2024-04-20T02:48:30.438 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 280 k 2024-04-20T02:48:30.438 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:48:30.439 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:48:30.439 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:48:30.494 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:48:30.503 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:48:30.616 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:48:30.664 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-fuse-18.2.2-1252.g15eecde2.el8.x86_64.rpm 422 kB/s | 95 kB 00:00 2024-04-20T02:48:30.664 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:48:30.664 INFO:teuthology.orchestra.run.smithi138.stdout:Total 419 kB/s | 95 kB 00:00 2024-04-20T02:48:30.665 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:48:30.675 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:48:30.675 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:48:30.730 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:48:30.739 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:48:30.779 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:48:30.853 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:48:31.071 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:48:31.083 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:48:31.322 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:48:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:48:42.278 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:48:42.325 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:48:42.326 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:48:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:48:42.878 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:48:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:48:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:48:42.879 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:48:42.927 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:48:42.927 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:48:42.927 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:48:42.927 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:48:42.927 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:48:43.036 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-mirror 2024-04-20T02:48:43.081 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-mirror 2024-04-20T02:48:43.399 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:43.441 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:48:45.108 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 4.1 kB 00:00 2024-04-20T02:48:45.228 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 38 kB/s | 4.1 kB 00:00 2024-04-20T02:48:45.482 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-20T02:48:45.553 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-20T02:48:47.832 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:48:47.832 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:48:47.832 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-04-20T02:48:47.832 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:48:47.832 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:48:47.832 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 3.5 M 2024-04-20T02:48:47.832 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:48:47.833 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:48:47.833 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:48:47.833 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-20T02:48:47.833 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:48:47.833 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 3.5 M 2024-04-20T02:48:47.833 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 16 M 2024-04-20T02:48:47.834 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:48:47.911 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:48:47.911 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:48:47.911 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-04-20T02:48:47.912 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:48:47.912 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:48:47.912 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 3.5 M 2024-04-20T02:48:47.912 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:48:47.912 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:48:47.912 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:48:47.912 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-04-20T02:48:47.912 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:48:47.913 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 3.5 M 2024-04-20T02:48:47.913 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 16 M 2024-04-20T02:48:47.913 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:48:48.389 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-mirror-18.2.2-1252.g15eecde2.el8.x86_64.rpm 6.2 MB/s | 3.5 MB 00:00 2024-04-20T02:48:48.390 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:48:48.390 INFO:teuthology.orchestra.run.smithi105.stdout:Total 6.2 MB/s | 3.5 MB 00:00 2024-04-20T02:48:48.390 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:48:48.403 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:48:48.403 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:48:48.472 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:48:48.480 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:48:48.705 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-mirror-18.2.2-1252.g15eecde2.el8.x86_64.rpm 4.4 MB/s | 3.5 MB 00:00 2024-04-20T02:48:48.706 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:48:48.706 INFO:teuthology.orchestra.run.smithi138.stdout:Total 4.4 MB/s | 3.5 MB 00:00 2024-04-20T02:48:48.706 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:48:48.718 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:48:48.718 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:48:48.788 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:48:48.797 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:48:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:48:49.136 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:48:49.254 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:48:49.511 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-mirror-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:48:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:48:49.932 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:49:00.558 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-mirror-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:49:00.558 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:49:01.155 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:01.156 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:49:01.156 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:49:01.156 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:01.156 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:49:01.323 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-nbd 2024-04-20T02:49:01.684 INFO:teuthology.orchestra.run.smithi138.stdout:Updating Subscription Management repositories. 2024-04-20T02:49:03.544 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 69 kB/s | 4.1 kB 00:00 2024-04-20T02:49:03.840 INFO:teuthology.orchestra.run.smithi138.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 57 kB/s | 4.5 kB 00:00 2024-04-20T02:49:04.177 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:49:04.177 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:49:04.734 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:04.735 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:49:04.735 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:49:04.735 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:04.735 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:49:04.909 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-nbd 2024-04-20T02:49:05.271 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2024-04-20T02:49:06.322 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-04-20T02:49:06.323 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:49:06.323 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-04-20T02:49:06.323 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:49:06.323 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-04-20T02:49:06.323 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 182 k 2024-04-20T02:49:06.323 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:06.323 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-04-20T02:49:06.323 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-04-20T02:49:06.324 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-04-20T02:49:06.324 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:06.324 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 182 k 2024-04-20T02:49:06.324 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 572 k 2024-04-20T02:49:06.324 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-04-20T02:49:06.510 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-nbd-18.2.2-1252.g15eecde2.el8.x86_64.rpm 979 kB/s | 182 kB 00:00 2024-04-20T02:49:06.511 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:49:06.511 INFO:teuthology.orchestra.run.smithi138.stdout:Total 973 kB/s | 182 kB 00:00 2024-04-20T02:49:06.511 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-04-20T02:49:06.522 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-04-20T02:49:06.522 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-04-20T02:49:06.578 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-04-20T02:49:06.587 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-04-20T02:49:06.732 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-04-20T02:49:06.940 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-nbd-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:49:07.209 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:49:07.294 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-20T02:49:07.668 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-20T02:49:10.090 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-04-20T02:49:10.090 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:49:10.091 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-04-20T02:49:10.091 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:49:10.091 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-04-20T02:49:10.091 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:18.2.2-1252.g15eecde2.el8 ceph 182 k 2024-04-20T02:49:10.091 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:10.091 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-04-20T02:49:10.091 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-04-20T02:49:10.091 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-04-20T02:49:10.091 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:10.092 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 182 k 2024-04-20T02:49:10.092 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 572 k 2024-04-20T02:49:10.092 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-04-20T02:49:10.368 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-nbd-18.2.2-1252.g15eecde2.el8.x86_64.rpm 658 kB/s | 182 kB 00:00 2024-04-20T02:49:10.369 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-04-20T02:49:10.369 INFO:teuthology.orchestra.run.smithi105.stdout:Total 655 kB/s | 182 kB 00:00 2024-04-20T02:49:10.369 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-04-20T02:49:10.381 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-04-20T02:49:10.381 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-04-20T02:49:10.439 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-04-20T02:49:10.448 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-04-20T02:49:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-04-20T02:49:10.799 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:49:11.062 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:49:20.196 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-nbd-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:49:20.197 INFO:teuthology.orchestra.run.smithi138.stdout:Installed products updated. 2024-04-20T02:49:20.846 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:20.846 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-04-20T02:49:20.846 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:49:20.846 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:20.846 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-04-20T02:49:21.048 DEBUG:teuthology.parallel:result is None 2024-04-20T02:49:24.180 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:18.2.2-1252.g15eecde2.el8.x86_64 1/1 2024-04-20T02:49:24.180 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2024-04-20T02:49:24.774 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:24.775 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-04-20T02:49:24.775 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:18.2.2-1252.g15eecde2.el8.x86_64 2024-04-20T02:49:24.775 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:24.775 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-04-20T02:49:24.929 DEBUG:teuthology.parallel:result is None 2024-04-20T02:49:24.929 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=15eecde2bab9803889d929c7e423d07970d8cbed 2024-04-20T02:49:25.151 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-20T02:49:25.174 INFO:teuthology.orchestra.run.smithi105.stdout:18.2.2-1252.g15eecde2.el8 2024-04-20T02:49:25.175 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1252.g15eecde2.el8 2024-04-20T02:49:25.175 INFO:teuthology.task.install:The correct ceph version 18.2.2-1252.g15eecde2 is installed. 2024-04-20T02:49:25.177 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=15eecde2bab9803889d929c7e423d07970d8cbed 2024-04-20T02:49:25.398 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-20T02:49:25.425 INFO:teuthology.orchestra.run.smithi138.stdout:18.2.2-1252.g15eecde2.el8 2024-04-20T02:49:25.425 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1252.g15eecde2.el8 2024-04-20T02:49:25.425 INFO:teuthology.task.install:The correct ceph version 18.2.2-1252.g15eecde2 is installed. 2024-04-20T02:49:25.427 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-20T02:49:25.428 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:25.428 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-20T02:49:25.458 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:25.458 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-20T02:49:25.498 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-20T02:49:25.498 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:25.499 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-04-20T02:49:25.526 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-20T02:49:25.592 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:25.593 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2024-04-20T02:49:25.623 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-20T02:49:25.692 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-20T02:49:25.692 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:25.692 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-20T02:49:25.719 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-20T02:49:25.786 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:25.786 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-20T02:49:25.814 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-20T02:49:25.885 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-20T02:49:25.886 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:25.886 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-04-20T02:49:25.912 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-20T02:49:25.979 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:25.979 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/stdin-killer 2024-04-20T02:49:26.007 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-20T02:49:26.075 INFO:teuthology.run_tasks:Running task ceph... 2024-04-20T02:49:26.157 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-20T02:49:26.157 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /var/log/ceph 2024-04-20T02:49:26.159 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /var/log/ceph 2024-04-20T02:49:26.185 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-20T02:49:26.185 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-20T02:49:26.225 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-20T02:49:26.251 INFO:tasks.ceph:Creating extra log directories... 2024-04-20T02:49:26.251 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-20T02:49:26.291 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-20T02:49:26.329 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-20T02:49:26.329 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-20T02:49:26.329 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272', 'branch': 'wip-yuri7-testing-2024-04-18-1351-reef', 'description': 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-snaps}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7664272', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-yuri7-testing-2024-04-18-1351-reef'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'min_in': 4}, 'workunit': {'branch': 'wip-yuri7-testing-2024-04-18-1351-reef', 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6867, 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri7-testing-2024-04-18-1351-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_15eecde2bab9803889d929c7e423d07970d8cbed/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '15eecde2bab9803889d929c7e423d07970d8cbed', 'targets': {'smithi105.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL1jsavjLcnnMFDvHQQVo4wyxEtGjfg3I1vllmWLMvSUwzstsJSDc7LOUC9yO94vHnJq4GyODKZtuW4kVlSj9eY=', 'smithi138.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAAMtHnh0m2QRjWnLyeuXiMSQwCODZQsnGwUbwKI81GUnu0f0TYE8o6NQJjkZG0RQWj/IBq/B8j+aAq1Y3zl+GQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'exec': {'client.0': ['sudo ceph osd pool create low_tier 4']}}, {'rados': {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'rollback': 10, 'set_chunk': 30, 'snap_create': 10, 'snap_remove': 10, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-20_01:10:46', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-20T02:49:26.330 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-20T02:49:26.358 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-20T02:49:26.375 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/run/ceph 2024-04-20T02:49:26.415 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/run/ceph 2024-04-20T02:49:26.442 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:26.442 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-04-20T02:49:26.496 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-20T02:49:26.496 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-04-20T02:49:26.552 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-20T02:49:26.552 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T02:49:26.552 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 175930 Links: 1 2024-04-20T02:49:26.553 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T02:49:26.553 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T02:49:26.553 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-20 02:48:49.300824418 +0000 2024-04-20T02:49:26.553 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-20 02:40:52.905476936 +0000 2024-04-20T02:49:26.553 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-20 02:40:52.905476936 +0000 2024-04-20T02:49:26.553 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2024-04-20T02:49:26.553 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-20T02:49:26.618 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-20T02:49:26.618 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-20T02:49:26.618 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000333845 s, 1.5 MB/s 2024-04-20T02:49:26.619 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-20T02:49:26.679 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-04-20T02:49:26.736 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-20T02:49:26.736 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T02:49:26.736 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 181637 Links: 1 2024-04-20T02:49:26.736 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T02:49:26.736 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T02:49:26.736 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-20 02:48:49.300824418 +0000 2024-04-20T02:49:26.736 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-20 02:40:53.254470137 +0000 2024-04-20T02:49:26.736 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-20 02:40:53.254470137 +0000 2024-04-20T02:49:26.737 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2024-04-20T02:49:26.737 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-20T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-20T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-20T02:49:26.802 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000240711 s, 2.1 MB/s 2024-04-20T02:49:26.803 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-20T02:49:26.861 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-04-20T02:49:26.919 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-20T02:49:26.919 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T02:49:26.919 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 181733 Links: 1 2024-04-20T02:49:26.919 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T02:49:26.919 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T02:49:26.919 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-20 02:48:49.301824398 +0000 2024-04-20T02:49:26.919 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-20 02:40:53.616463086 +0000 2024-04-20T02:49:26.919 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-20 02:40:53.616463086 +0000 2024-04-20T02:49:26.920 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2024-04-20T02:49:26.920 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-20T02:49:26.984 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-20T02:49:26.984 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-20T02:49:26.984 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000164073 s, 3.1 MB/s 2024-04-20T02:49:26.985 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-20T02:49:27.042 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-04-20T02:49:27.100 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-20T02:49:27.100 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T02:49:27.100 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 178555 Links: 1 2024-04-20T02:49:27.100 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T02:49:27.100 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T02:49:27.100 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-04-20 02:48:49.301824398 +0000 2024-04-20T02:49:27.100 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-04-20 02:40:53.958456424 +0000 2024-04-20T02:49:27.100 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-04-20 02:40:53.958456424 +0000 2024-04-20T02:49:27.100 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2024-04-20T02:49:27.101 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-20T02:49:27.165 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-04-20T02:49:27.165 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-04-20T02:49:27.165 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000224737 s, 2.3 MB/s 2024-04-20T02:49:27.166 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-20T02:49:27.225 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-20T02:49:27.225 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:27.225 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2024-04-20T02:49:27.246 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-20T02:49:27.246 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 2024-04-20T02:49:27.306 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-20T02:49:27.307 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T02:49:27.307 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 181648 Links: 1 2024-04-20T02:49:27.307 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T02:49:27.307 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T02:49:27.307 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-04-20 02:48:49.683260095 +0000 2024-04-20T02:49:27.307 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-04-20 02:40:52.907355544 +0000 2024-04-20T02:49:27.307 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-04-20 02:40:52.907355544 +0000 2024-04-20T02:49:27.307 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-04-20T02:49:27.307 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-20T02:49:27.377 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-04-20T02:49:27.377 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-04-20T02:49:27.378 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000258014 s, 2.0 MB/s 2024-04-20T02:49:27.379 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-20T02:49:27.440 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 2024-04-20T02:49:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-20T02:49:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T02:49:27.500 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 182529 Links: 1 2024-04-20T02:49:27.500 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T02:49:27.500 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T02:49:27.500 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-04-20 02:48:49.683260095 +0000 2024-04-20T02:49:27.500 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-04-20 02:40:53.254348684 +0000 2024-04-20T02:49:27.501 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-04-20 02:40:53.254348684 +0000 2024-04-20T02:49:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-04-20T02:49:27.501 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-20T02:49:27.568 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-04-20T02:49:27.568 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-04-20T02:49:27.568 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000313273 s, 1.6 MB/s 2024-04-20T02:49:27.570 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-20T02:49:27.629 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 2024-04-20T02:49:27.688 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-20T02:49:27.688 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T02:49:27.688 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 182620 Links: 1 2024-04-20T02:49:27.688 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T02:49:27.688 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T02:49:27.688 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-04-20 02:48:49.683260095 +0000 2024-04-20T02:49:27.688 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-04-20 02:40:53.629341271 +0000 2024-04-20T02:49:27.689 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-04-20 02:40:53.629341271 +0000 2024-04-20T02:49:27.689 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-04-20T02:49:27.689 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-20T02:49:27.755 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-04-20T02:49:27.756 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-04-20T02:49:27.756 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000195923 s, 2.6 MB/s 2024-04-20T02:49:27.758 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-20T02:49:27.816 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 2024-04-20T02:49:27.874 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-20T02:49:27.874 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-20T02:49:27.874 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 6h/6d Inode: 174881 Links: 1 2024-04-20T02:49:27.874 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-20T02:49:27.874 INFO:teuthology.orchestra.run.smithi138.stdout:Context: system_u:object_r:device_t:s0 2024-04-20T02:49:27.874 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2024-04-20 02:48:49.683260095 +0000 2024-04-20T02:49:27.875 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2024-04-20 02:40:53.974334450 +0000 2024-04-20T02:49:27.875 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2024-04-20 02:40:53.974334450 +0000 2024-04-20T02:49:27.875 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2024-04-20T02:49:27.875 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-20T02:49:27.941 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2024-04-20T02:49:27.942 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2024-04-20T02:49:27.942 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000231924 s, 2.2 MB/s 2024-04-20T02:49:27.943 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-20T02:49:28.001 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-20T02:49:28.001 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-20T02:49:28.001 INFO:tasks.ceph:Generating config... 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-20T02:49:28.003 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd backoff on peering = True 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd max scrubs = 3 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-20T02:49:28.004 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-20T02:49:28.005 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-20T02:49:28.005 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-20T02:49:28.005 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-20T02:49:28.005 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-20T02:49:28.005 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-20T02:49:28.005 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-04-20T02:49:28.005 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-20T02:49:28.005 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-04-20T02:49:28.005 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-04-20T02:49:28.005 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-04-20T02:49:28.005 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-04-20T02:49:28.005 INFO:tasks.ceph:[global] ms tcp read timeout = 5 2024-04-20T02:49:28.005 INFO:tasks.ceph:[global] ms type = async 2024-04-20T02:49:28.005 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-04-20T02:49:28.005 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 2 2024-04-20T02:49:28.005 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 1 2024-04-20T02:49:28.006 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-04-20T02:49:28.006 INFO:tasks.ceph:[global] osd_pg_log_trim_min = 0 2024-04-20T02:49:28.006 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-04-20T02:49:28.006 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-04-20T02:49:28.006 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-20T02:49:28.006 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-20T02:49:28.006 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-20T02:49:28.006 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-20T02:49:28.006 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-20T02:49:28.006 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-20T02:49:28.006 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-20T02:49:28.006 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-20T02:49:28.006 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-20T02:49:28.006 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-20T02:49:28.007 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-20T02:49:28.007 INFO:tasks.ceph:Setting up mon.a... 2024-04-20T02:49:28.007 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-20T02:49:28.070 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/ceph.keyring 2024-04-20T02:49:28.074 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-20T02:49:28.121 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-20T02:49:28.149 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.105'), ('mon.c', '[v2:172.21.15.105:3301,v1:172.21.15.105:6790]'), ('mon.b', '172.21.15.138')] 2024-04-20T02:49:28.149 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.105,[v2:172.21.15.105:3301,v1:172.21.15.105:6790],172.21.15.138', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-20T02:49:28.150 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:28.150 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-20T02:49:28.206 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.105 --addv c '[v2:172.21.15.105:3301,v1:172.21.15.105:6790]' --add b 172.21.15.138 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:28.296 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:28.296 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: generated fsid a524a365-eaec-4ebd-aa4d-0d76d55a252f 2024-04-20T02:49:28.296 INFO:teuthology.orchestra.run.smithi105.stdout:setting min_mon_release = pacific 2024-04-20T02:49:28.296 INFO:teuthology.orchestra.run.smithi105.stdout:epoch 0 2024-04-20T02:49:28.296 INFO:teuthology.orchestra.run.smithi105.stdout:fsid a524a365-eaec-4ebd-aa4d-0d76d55a252f 2024-04-20T02:49:28.296 INFO:teuthology.orchestra.run.smithi105.stdout:last_changed 2024-04-20T02:49:28.297285+0000 2024-04-20T02:49:28.296 INFO:teuthology.orchestra.run.smithi105.stdout:created 2024-04-20T02:49:28.297285+0000 2024-04-20T02:49:28.296 INFO:teuthology.orchestra.run.smithi105.stdout:min_mon_release 16 (pacific) 2024-04-20T02:49:28.296 INFO:teuthology.orchestra.run.smithi105.stdout:election_strategy: 3 2024-04-20T02:49:28.297 INFO:teuthology.orchestra.run.smithi105.stdout:0: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.a 2024-04-20T02:49:28.297 INFO:teuthology.orchestra.run.smithi105.stdout:1: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.b 2024-04-20T02:49:28.297 INFO:teuthology.orchestra.run.smithi105.stdout:2: [v2:172.21.15.105:3301/0,v1:172.21.15.105:6790/0] mon.c 2024-04-20T02:49:28.297 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-20T02:49:28.298 DEBUG:teuthology.orchestra.run.smithi105:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-20T02:49:28.315 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a524a365-eaec-4ebd-aa4d-0d76d55a252f... 2024-04-20T02:49:28.315 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-20T02:49:28.357 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-20T02:49:28.400 INFO:teuthology.orchestra.run.smithi105.stdout:[global] 2024-04-20T02:49:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: chdir = "" 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: auth supported = cephx 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: filestore xattr use omap = true 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: mon clock drift allowed = 1.000 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: osd crush chooseleaf type = 0 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: auth debug = true 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on old message = true 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on bug = true 2024-04-20T02:49:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: mon pg warn max object skew = 0 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: # disable pg_autoscaler by default for new pools 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default size = 2 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow primary affinity = true 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow pg remap = true 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on legacy crush tunables = false 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on crush straw calc version zero = false 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on no sortbitwise = false 2024-04-20T02:49:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on osd down out interval zero = false 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on too few osds = false 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: mon_allow_pool_size_one = true 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: osd default data pool replay window = 5 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: mon allow pool delete = true 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: mon cluster log file level = debug 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: debug asserts on shutdown = true 2024-04-20T02:49:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: mon health detail to clog = false 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: mon host = "172.21.15.105,[v2:172.21.15.105:3301,v1:172.21.15.105:6790],172.21.15.138" 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: mon client directed command retry = 5 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: mon election default strategy = 3 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: ms bind msgr1 = False 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: ms bind msgr2 = True 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject socket failures = 2500 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: ms tcp read timeout = 5 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: ms type = async 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: osd_async_recovery_min_cost = 1 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: osd_max_pg_log_entries = 2 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: osd_min_pg_log_entries = 1 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: osd_pg_log_trim_min = 0 2024-04-20T02:49:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: osd_pool_default_min_size = 2 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: osd_pool_default_size = 3 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: fsid = a524a365-eaec-4ebd-aa4d-0d76d55a252f 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout:[osd] 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: osd journal size = 100 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub load threshold = 5.0 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub max interval = 120 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: osd mclock profile = high_recovery_ops 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: osd recover clone overlap = true 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: osd recovery max chunk = 1048576 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug shutdown = true 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug op order = true 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify stray on activate = true 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: osd open classes on start = true 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug pg log writeout = true 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: osd deep scrub update digest min age = 30 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: osd map max advance = 10 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: journal zero on create = true 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: filestore ondisk finisher threads = 3 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: filestore apply finisher threads = 3 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: bdev debug aio = true 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug misdirected ops = true 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: debug monc = 20 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore block size = 96636764160 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore compression algorithm = zlib 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore compression mode = aggressive 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore fsck on mount = True 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore zero block detection = True 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluefs = 20 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluestore = 20 2024-04-20T02:49:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: debug osd = 20 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: debug rocksdb = 10 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd backfillfull_ratio = 0.85 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd full ratio = 0.9 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd nearfull ratio = 0.8 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: osd backoff on peering = True 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: osd blocked scrub grace period = 3600 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug reject backfill probability = 0.3 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify cached snaps = True 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify missing on start = True 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: osd delete sleep = 1 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: osd failsafe full ratio = 0.95 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: osd max backfills = 3 2024-04-20T02:49:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: osd max markdown count = 1000 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: osd max scrubs = 3 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: osd mclock override recovery settings = True 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: osd objectstore = bluestore 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: osd op queue = debug_random 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: osd op queue cut off = debug_random 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub min interval = 60 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: osd shutdown pgref assert = True 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: osd snap trim sleep = 2 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr] 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: debug mgr = 20 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2024-04-20T02:49:28.409 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: mgr/telemetry/nag = false 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout:[mon] 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: debug paxos = 20 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: mon data avail warn = 5 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: mon mgr mkfs grace = 240 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd reporter subtree level = osd 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd prime pg temp = true 2024-04-20T02:49:28.410 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: auth mon ticket ttl = 660 # 11m 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: auth service ticket ttl = 240 # 4m 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: # don't complain about insecure global_id in the test suite 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: # 1m isn't quite enough 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: mon_down_mkfs_grace = 2m 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_filestore_osds = false 2024-04-20T02:49:28.411 INFO:teuthology.orchestra.run.smithi105.stdout: mon min osdmap epochs = 50 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: mon osdmap full prune interval = 2 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: mon osdmap full prune min = 15 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: mon osdmap full prune txsize = 2 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: mon scrub interval = 300 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: paxos service trim min = 10 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout:[client] 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: rgw cache enabled = true 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable ops log = true 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable usage log = true 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.a] 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.c] 2024-04-20T02:49:28.412 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.b] 2024-04-20T02:49:28.413 INFO:teuthology.orchestra.run.smithi138.stdout:[global] 2024-04-20T02:49:28.413 INFO:teuthology.orchestra.run.smithi138.stdout: chdir = "" 2024-04-20T02:49:28.413 INFO:teuthology.orchestra.run.smithi138.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-20T02:49:28.413 INFO:teuthology.orchestra.run.smithi138.stdout: auth supported = cephx 2024-04-20T02:49:28.413 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: filestore xattr use omap = true 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: mon clock drift allowed = 1.000 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: osd crush chooseleaf type = 0 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: auth debug = true 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on old message = true 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on bug = true 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: mon pg warn max object skew = 0 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.414 INFO:teuthology.orchestra.run.smithi138.stdout: # disable pg_autoscaler by default for new pools 2024-04-20T02:49:28.415 INFO:teuthology.orchestra.run.smithi138.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-20T02:49:28.415 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.415 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default size = 2 2024-04-20T02:49:28.415 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.415 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow primary affinity = true 2024-04-20T02:49:28.415 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow pg remap = true 2024-04-20T02:49:28.415 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on legacy crush tunables = false 2024-04-20T02:49:28.415 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on crush straw calc version zero = false 2024-04-20T02:49:28.415 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on no sortbitwise = false 2024-04-20T02:49:28.417 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on osd down out interval zero = false 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on too few osds = false 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: mon_allow_pool_size_one = true 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: osd default data pool replay window = 5 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: mon allow pool delete = true 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: mon cluster log file level = debug 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: debug asserts on shutdown = true 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: mon health detail to clog = false 2024-04-20T02:49:28.418 INFO:teuthology.orchestra.run.smithi138.stdout: mon host = "172.21.15.105,[v2:172.21.15.105:3301,v1:172.21.15.105:6790],172.21.15.138" 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: mon client directed command retry = 5 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: mon election default strategy = 3 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: ms bind msgr1 = False 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: ms bind msgr2 = True 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: ms inject socket failures = 2500 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: ms tcp read timeout = 5 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: ms type = async 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: osd_async_recovery_min_cost = 1 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: osd_max_pg_log_entries = 2 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: osd_min_pg_log_entries = 1 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: osd_pg_log_trim_min = 0 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: osd_pool_default_min_size = 2 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: osd_pool_default_size = 3 2024-04-20T02:49:28.419 INFO:teuthology.orchestra.run.smithi138.stdout: fsid = a524a365-eaec-4ebd-aa4d-0d76d55a252f 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout:[osd] 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: osd journal size = 100 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub load threshold = 5.0 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub max interval = 120 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: osd mclock profile = high_recovery_ops 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: osd recover clone overlap = true 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: osd recovery max chunk = 1048576 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug shutdown = true 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug op order = true 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify stray on activate = true 2024-04-20T02:49:28.420 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: osd open classes on start = true 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug pg log writeout = true 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: osd deep scrub update digest min age = 30 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: osd map max advance = 10 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: journal zero on create = true 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: filestore ondisk finisher threads = 3 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: filestore apply finisher threads = 3 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: bdev debug aio = true 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug misdirected ops = true 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: debug monc = 20 2024-04-20T02:49:28.421 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore block size = 96636764160 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore compression algorithm = zlib 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore compression mode = aggressive 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore fsck on mount = True 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore zero block detection = True 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: debug bluefs = 20 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: debug bluestore = 20 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: debug osd = 20 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: debug rocksdb = 10 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd backfillfull_ratio = 0.85 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd full ratio = 0.9 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd nearfull ratio = 0.8 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: osd backoff on peering = True 2024-04-20T02:49:28.422 INFO:teuthology.orchestra.run.smithi138.stdout: osd blocked scrub grace period = 3600 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug reject backfill probability = 0.3 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify cached snaps = True 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify missing on start = True 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd delete sleep = 1 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd failsafe full ratio = 0.95 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd max backfills = 3 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd max markdown count = 1000 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd max scrubs = 3 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd mclock override recovery settings = True 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd objectstore = bluestore 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd op queue = debug_random 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd op queue cut off = debug_random 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub min interval = 60 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd shutdown pgref assert = True 2024-04-20T02:49:28.423 INFO:teuthology.orchestra.run.smithi138.stdout: osd snap trim sleep = 2 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr] 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: debug mgr = 20 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: mgr/telemetry/nag = false 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout:[mon] 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: debug paxos = 20 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2024-04-20T02:49:28.424 INFO:teuthology.orchestra.run.smithi138.stdout: mon data avail warn = 5 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: mon mgr mkfs grace = 240 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd reporter subtree level = osd 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd prime pg temp = true 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: auth mon ticket ttl = 660 # 11m 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: auth service ticket ttl = 240 # 4m 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: # don't complain about insecure global_id in the test suite 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.425 INFO:teuthology.orchestra.run.smithi138.stdout: # 1m isn't quite enough 2024-04-20T02:49:28.426 INFO:teuthology.orchestra.run.smithi138.stdout: mon_down_mkfs_grace = 2m 2024-04-20T02:49:28.426 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.426 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_filestore_osds = false 2024-04-20T02:49:28.426 INFO:teuthology.orchestra.run.smithi138.stdout: mon min osdmap epochs = 50 2024-04-20T02:49:28.426 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune interval = 2 2024-04-20T02:49:28.426 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune min = 15 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout: mon osdmap full prune txsize = 2 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout: mon scrub interval = 300 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout: paxos service trim min = 10 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout:[client] 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout: rgw cache enabled = true 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable ops log = true 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable usage log = true 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.a] 2024-04-20T02:49:28.427 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.c] 2024-04-20T02:49:28.428 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.b] 2024-04-20T02:49:28.429 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-20T02:49:28.429 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-20T02:49:28.528 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-20T02:49:28.528 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:28.528 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-20T02:49:28.544 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:28.545 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-20T02:49:28.601 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:49:28.601 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:28.601 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-20T02:49:28.602 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-20T02:49:28.678 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:28.678 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:28.734 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:49:28.735 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:28.735 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-20T02:49:28.735 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-20T02:49:28.773 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:28.773 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:28.830 INFO:tasks.ceph:Setting up mon nodes... 2024-04-20T02:49:28.830 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-20T02:49:28.831 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-20T02:49:28.884 INFO:teuthology.orchestra.run.smithi105.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-20T02:49:28.887 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-20T02:49:28.992 INFO:teuthology.orchestra.run.smithi138.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-20T02:49:28.995 INFO:tasks.ceph:Setting up mds nodes... 2024-04-20T02:49:28.995 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-20T02:49:28.995 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-20T02:49:29.039 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-20T02:49:29.052 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-20T02:49:29.096 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-20T02:49:29.109 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-20T02:49:29.109 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-20T02:49:29.109 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-20T02:49:29.136 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-20T02:49:29.136 INFO:tasks.ceph:role: osd.0 2024-04-20T02:49:29.136 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:49:29.137 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-20T02:49:29.211 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T02:49:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T02:49:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T02:49:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T02:49:29.211 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T02:49:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-04-20T02:49:29.211 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T02:49:29.211 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T02:49:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T02:49:29.211 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T02:49:29.639 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-04-20T02:49:29.641 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-04-20T02:49:29.641 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-20T02:49:29.822 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-20T02:49:29.855 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-20T02:49:29.922 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-20T02:49:29.923 INFO:tasks.ceph:role: osd.1 2024-04-20T02:49:29.923 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:49:29.923 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-20T02:49:29.996 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T02:49:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T02:49:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T02:49:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T02:49:29.997 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T02:49:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-04-20T02:49:29.997 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T02:49:29.997 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T02:49:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T02:49:29.997 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T02:49:30.426 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-04-20T02:49:30.427 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-04-20T02:49:30.428 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-20T02:49:30.536 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-20T02:49:30.566 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-20T02:49:30.633 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-20T02:49:30.633 INFO:tasks.ceph:role: osd.2 2024-04-20T02:49:30.633 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:49:30.633 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-20T02:49:30.707 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T02:49:30.707 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T02:49:30.707 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T02:49:30.708 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T02:49:30.708 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T02:49:30.708 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-04-20T02:49:30.708 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T02:49:30.708 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T02:49:30.708 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T02:49:30.708 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T02:49:31.156 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-04-20T02:49:31.158 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-04-20T02:49:31.158 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-20T02:49:31.326 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-20T02:49:31.360 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-20T02:49:31.428 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-20T02:49:31.429 INFO:tasks.ceph:role: osd.3 2024-04-20T02:49:31.429 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:49:31.429 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-20T02:49:31.504 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T02:49:31.504 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T02:49:31.505 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T02:49:31.505 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T02:49:31.505 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T02:49:31.505 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2024-04-20T02:49:31.505 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T02:49:31.505 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T02:49:31.506 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T02:49:31.506 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T02:49:31.931 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2024-04-20T02:49:31.932 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2024-04-20T02:49:31.932 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-20T02:49:32.113 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-20T02:49:32.147 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:32.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:32.236+0000 7ff04e4f17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-20T02:49:32.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:32.237+0000 7ff04e4f17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-20T02:49:32.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:32.237+0000 7ff04e4f17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-20T02:49:32.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:32.237+0000 7ff04e4f17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-20T02:49:32.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:32.237+0000 7ff04e4f17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-20T02:49:32.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:32.237+0000 7ff04e4f17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-20T02:49:35.091 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-20T02:49:35.124 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:35.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:35.217+0000 7fd57397e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-20T02:49:35.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:35.218+0000 7fd57397e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-20T02:49:35.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:35.218+0000 7fd57397e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-20T02:49:35.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:35.218+0000 7fd57397e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-20T02:49:35.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:35.218+0000 7fd57397e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-20T02:49:35.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:35.218+0000 7fd57397e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-20T02:49:38.071 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-20T02:49:38.103 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:38.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:38.196+0000 7f1c351957c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-20T02:49:38.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:38.196+0000 7f1c351957c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-20T02:49:38.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:38.196+0000 7f1c351957c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-20T02:49:38.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:38.196+0000 7f1c351957c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-20T02:49:38.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:38.196+0000 7f1c351957c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-20T02:49:38.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:38.197+0000 7f1c351957c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-20T02:49:41.045 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-20T02:49:41.077 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:41.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:41.170+0000 7f016b24e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-20T02:49:41.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:41.170+0000 7f016b24e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-20T02:49:41.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:41.170+0000 7f016b24e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-20T02:49:41.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:41.170+0000 7f016b24e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-20T02:49:41.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:41.170+0000 7f016b24e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-20T02:49:41.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:49:41.171+0000 7f016b24e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-20T02:49:44.021 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-20T02:49:44.066 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-20T02:49:44.112 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-20T02:49:44.112 INFO:tasks.ceph:role: osd.4 2024-04-20T02:49:44.112 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:49:44.112 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-20T02:49:44.149 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T02:49:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T02:49:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T02:49:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T02:49:44.150 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T02:49:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-04-20T02:49:44.150 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T02:49:44.150 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T02:49:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T02:49:44.150 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T02:49:44.582 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-04-20T02:49:44.584 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-04-20T02:49:44.585 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-20T02:49:44.761 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-20T02:49:44.799 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-20T02:49:44.870 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-20T02:49:44.870 INFO:tasks.ceph:role: osd.5 2024-04-20T02:49:44.870 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:49:44.870 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-20T02:49:44.948 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T02:49:44.949 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T02:49:44.949 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T02:49:44.949 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T02:49:44.949 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T02:49:44.949 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-04-20T02:49:44.949 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T02:49:44.949 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T02:49:44.949 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T02:49:44.949 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T02:49:45.370 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-04-20T02:49:45.372 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-04-20T02:49:45.372 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-20T02:49:45.485 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-20T02:49:45.519 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-20T02:49:45.588 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-20T02:49:45.588 INFO:tasks.ceph:role: osd.6 2024-04-20T02:49:45.589 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:49:45.589 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-20T02:49:45.663 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T02:49:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T02:49:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T02:49:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T02:49:45.664 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T02:49:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-04-20T02:49:45.664 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T02:49:45.664 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T02:49:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T02:49:45.664 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T02:49:46.101 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-04-20T02:49:46.103 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-04-20T02:49:46.103 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-20T02:49:46.245 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-20T02:49:46.278 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-20T02:49:46.347 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-20T02:49:46.347 INFO:tasks.ceph:role: osd.7 2024-04-20T02:49:46.347 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:49:46.349 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-20T02:49:46.423 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-20T02:49:46.424 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-20T02:49:46.424 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-20T02:49:46.424 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-20T02:49:46.424 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-20T02:49:46.424 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2024-04-20T02:49:46.424 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-20T02:49:46.424 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-20T02:49:46.424 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-20T02:49:46.424 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-20T02:49:46.867 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2024-04-20T02:49:46.869 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2024-04-20T02:49:46.869 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-20T02:49:47.051 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-20T02:49:47.085 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:47.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:47.175+0000 7fc75a2cb7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-20T02:49:47.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:47.175+0000 7fc75a2cb7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-20T02:49:47.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:47.175+0000 7fc75a2cb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-20T02:49:47.175 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:47.175+0000 7fc75a2cb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-20T02:49:47.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:47.175+0000 7fc75a2cb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-20T02:49:47.176 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:47.176+0000 7fc75a2cb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-20T02:49:50.027 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-20T02:49:50.059 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:50.151 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:50.151+0000 7f39911c47c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-20T02:49:50.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:50.151+0000 7f39911c47c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-20T02:49:50.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:50.152+0000 7f39911c47c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-20T02:49:50.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:50.152+0000 7f39911c47c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-20T02:49:50.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:50.152+0000 7f39911c47c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-20T02:49:50.152 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:50.152+0000 7f39911c47c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-20T02:49:52.999 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-20T02:49:53.029 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:53.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:53.121+0000 7f3d12dcd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-20T02:49:53.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:53.121+0000 7f3d12dcd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-20T02:49:53.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:53.121+0000 7f3d12dcd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-20T02:49:53.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:53.121+0000 7f3d12dcd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-20T02:49:53.121 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:53.121+0000 7f3d12dcd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-20T02:49:53.122 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:53.122+0000 7f3d12dcd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-20T02:49:55.972 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-20T02:49:56.003 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:49:56.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:56.095+0000 7fa3fe5757c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-20T02:49:56.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:56.095+0000 7fa3fe5757c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-20T02:49:56.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:56.095+0000 7fa3fe5757c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-20T02:49:56.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:56.095+0000 7fa3fe5757c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-20T02:49:56.096 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:56.095+0000 7fa3fe5757c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-20T02:49:56.097 INFO:teuthology.orchestra.run.smithi138.stderr:2024-04-20T02:49:56.096+0000 7fa3fe5757c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-20T02:49:58.945 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-20T02:49:58.977 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-20T02:49:58.977 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:58.978 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-20T02:49:59.011 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:59.011 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-20T02:49:59.082 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:59.083 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-20T02:49:59.152 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:59.152 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-20T02:49:59.220 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:59.220 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-20T02:49:59.287 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:59.287 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-20T02:49:59.320 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:59.320 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-20T02:49:59.389 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:59.390 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-20T02:49:59.459 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:59.459 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-20T02:49:59.529 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:59.529 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-20T02:49:59.598 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:49:59.598 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-20T02:49:59.616 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:49:59.616 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-20T02:49:59.656 INFO:tasks.ceph:Adding keys to all mons... 2024-04-20T02:49:59.657 DEBUG:teuthology.orchestra.run.smithi105:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-20T02:49:59.661 DEBUG:teuthology.orchestra.run.smithi138:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-20T02:49:59.699 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr.y] 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQA4LSNmzc3ONBAAilRtfumawvxWTEZ+gqnP4Q== 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.0] 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQA8LSNmiVgvDhAAvNplACVpviI7lnwvzVZ9rw== 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.1] 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQA/LSNm5jwODRAA+PznRVuey3VBd5Vlc8OYUw== 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.2] 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQBCLSNmeBbGCxAAjXTFxCPHhnwssoSyq4KEQQ== 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.3] 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQBFLSNmOPo6ChAAobAQ7/ZxndWyq3oWq0eW1A== 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr.x] 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQA4LSNmLOU8OxAA419S8ZwY4wvIB1q25cxoQg== 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.4] 2024-04-20T02:49:59.700 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQBLLSNm+J6CChAAHs8yO+Ty26EVT+E9sSPhyw== 2024-04-20T02:49:59.701 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.5] 2024-04-20T02:49:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQBOLSNmTlcXCRAAXjctER/CBW6DxZ3gjK6dQA== 2024-04-20T02:49:59.701 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.6] 2024-04-20T02:49:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQBRLSNmJANHBxAAZN0FbbitZEl8bU1dhL2MLA== 2024-04-20T02:49:59.701 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.7] 2024-04-20T02:49:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQBULSNmB7HJBRAAaKfPrDHC1mq30GAcg7JWYw== 2024-04-20T02:49:59.701 INFO:teuthology.orchestra.run.smithi105.stdout:[client.0] 2024-04-20T02:49:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQA5LSNmRcZoAhAAmzu6j4n4yKA2Ej5yHsiXXw== 2024-04-20T02:49:59.701 INFO:teuthology.orchestra.run.smithi105.stdout:[client.1] 2024-04-20T02:49:59.701 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQA5LSNmW0POBRAAT07tuJqmx98tbN2Y0elshQ== 2024-04-20T02:49:59.723 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr.y] 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQA4LSNmzc3ONBAAilRtfumawvxWTEZ+gqnP4Q== 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.0] 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQA8LSNmiVgvDhAAvNplACVpviI7lnwvzVZ9rw== 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.1] 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQA/LSNm5jwODRAA+PznRVuey3VBd5Vlc8OYUw== 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.2] 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBCLSNmeBbGCxAAjXTFxCPHhnwssoSyq4KEQQ== 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.3] 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBFLSNmOPo6ChAAobAQ7/ZxndWyq3oWq0eW1A== 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr.x] 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQA4LSNmLOU8OxAA419S8ZwY4wvIB1q25cxoQg== 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.4] 2024-04-20T02:49:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBLLSNm+J6CChAAHs8yO+Ty26EVT+E9sSPhyw== 2024-04-20T02:49:59.725 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.5] 2024-04-20T02:49:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBOLSNmTlcXCRAAXjctER/CBW6DxZ3gjK6dQA== 2024-04-20T02:49:59.725 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.6] 2024-04-20T02:49:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBRLSNmJANHBxAAZN0FbbitZEl8bU1dhL2MLA== 2024-04-20T02:49:59.725 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.7] 2024-04-20T02:49:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQBULSNmB7HJBRAAaKfPrDHC1mq30GAcg7JWYw== 2024-04-20T02:49:59.725 INFO:teuthology.orchestra.run.smithi138.stdout:[client.0] 2024-04-20T02:49:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQA5LSNmRcZoAhAAmzu6j4n4yKA2Ej5yHsiXXw== 2024-04-20T02:49:59.725 INFO:teuthology.orchestra.run.smithi138.stdout:[client.1] 2024-04-20T02:49:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQA5LSNmW0POBRAAT07tuJqmx98tbN2Y0elshQ== 2024-04-20T02:49:59.726 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-20T02:49:59.745 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-20T02:49:59.822 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:49:59.824 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:49:59.879 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:49:59.881 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:49:59.935 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:49:59.937 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:49:59.991 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:49:59.993 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:50:00.048 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-20T02:50:00.050 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-20T02:50:00.104 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:50:00.106 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:50:00.159 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:50:00.161 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:50:00.214 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:50:00.217 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:50:00.271 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:50:00.273 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-20T02:50:00.326 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-20T02:50:00.329 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-20T02:50:00.385 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-20T02:50:00.388 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-20T02:50:00.444 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-20T02:50:00.444 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-20T02:50:00.473 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-20T02:50:00.571 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-20T02:50:00.598 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-20T02:50:00.665 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-20T02:50:00.764 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-20T02:50:00.790 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-20T02:50:00.819 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-20T02:50:00.919 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-20T02:50:00.947 DEBUG:teuthology.orchestra.run.smithi105:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:50:00.949 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-20T02:50:01.004 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-20T02:50:01.005 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-20T02:50:01.005 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-20T02:50:01.008 INFO:tasks.ceph.mon.a:Started 2024-04-20T02:50:01.008 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-20T02:50:01.008 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-20T02:50:01.010 INFO:tasks.ceph.mon.c:Started 2024-04-20T02:50:01.010 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-20T02:50:01.010 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-20T02:50:01.046 INFO:tasks.ceph.mon.b:Started 2024-04-20T02:50:01.046 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-20T02:50:01.046 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-20T02:50:01.047 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-20T02:50:01.050 INFO:tasks.ceph.mgr.y:Started 2024-04-20T02:50:01.050 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-20T02:50:01.050 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-20T02:50:01.052 INFO:tasks.ceph.mgr.x:Started 2024-04-20T02:50:01.052 INFO:tasks.ceph:Setting crush tunables to default 2024-04-20T02:50:01.052 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd crush tunables default 2024-04-20T02:50:07.471 INFO:teuthology.orchestra.run.smithi105.stderr:adjusted tunables profile to default 2024-04-20T02:50:07.483 INFO:tasks.ceph:check_enable_crimson: False 2024-04-20T02:50:07.483 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-20T02:50:07.483 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:50:07.483 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-20T02:50:07.511 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:50:07.511 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-20T02:50:07.578 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:50:07.578 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-20T02:50:07.645 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-04-20T02:50:07.645 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-20T02:50:07.712 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:50:07.712 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-20T02:50:07.740 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:50:07.740 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-20T02:50:07.807 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:50:07.807 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-20T02:50:07.821 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:07.821+0000 7f609f66b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-20T02:50:07.846 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-04-20T02:50:07.847 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-20T02:50:07.903 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:07.904+0000 7f609f66b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-20T02:50:07.913 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new 8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6 0 2024-04-20T02:50:07.980 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:07.981+0000 7f609f66b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-20T02:50:08.054 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:08.054+0000 7f609f66b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-20T02:50:08.147 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:08.147+0000 7f0bfac6c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-20T02:50:08.174 INFO:teuthology.orchestra.run.smithi138.stdout:0 2024-04-20T02:50:08.187 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new c5ae8a64-fc7c-47ab-af6a-4b762a840645 1 2024-04-20T02:50:08.227 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:08.228+0000 7f0bfac6c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-20T02:50:08.236 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:08.236+0000 7f609f66b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-20T02:50:08.301 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:08.301+0000 7f0bfac6c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-20T02:50:08.371 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:08.371+0000 7f0bfac6c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-20T02:50:08.409 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:08.410+0000 7f609f66b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-20T02:50:08.448 INFO:teuthology.orchestra.run.smithi138.stdout:1 2024-04-20T02:50:08.462 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new 336e852e-5fa6-4ac0-b053-d31ad40d7c4e 2 2024-04-20T02:50:08.526 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:08.527+0000 7f609f66b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-20T02:50:08.543 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:08.543+0000 7f0bfac6c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-20T02:50:08.628 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:08.628+0000 7f609f66b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-20T02:50:08.706 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:08.707+0000 7f609f66b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-20T02:50:08.709 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:08.709+0000 7f0bfac6c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-20T02:50:08.721 INFO:teuthology.orchestra.run.smithi138.stdout:2 2024-04-20T02:50:08.734 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new 695cab3a-74d8-4046-ac61-2656dc0d893e 3 2024-04-20T02:50:08.818 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:08.818+0000 7f0bfac6c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-20T02:50:08.911 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:08.911+0000 7f0bfac6c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-20T02:50:08.936 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:08.937+0000 7f609f66b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-20T02:50:08.988 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:08.988+0000 7f0bfac6c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-20T02:50:09.010 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:09.010+0000 7f609f66b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-20T02:50:09.028 INFO:teuthology.orchestra.run.smithi138.stdout:3 2024-04-20T02:50:09.045 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new 1d41da44-d894-40ac-acaa-aa0df15296c6 4 2024-04-20T02:50:09.216 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:09.216+0000 7f0bfac6c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-20T02:50:09.266 INFO:teuthology.orchestra.run.smithi138.stdout:4 2024-04-20T02:50:09.279 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new 3e1c33fa-6c8c-4898-99bc-462ecef25f30 5 2024-04-20T02:50:09.287 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:09.287+0000 7f0bfac6c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-20T02:50:09.428 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:09.428+0000 7f609f66b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-20T02:50:09.498 INFO:teuthology.orchestra.run.smithi138.stdout:5 2024-04-20T02:50:09.510 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new fd1fc124-7cf8-4628-9076-1f27d6385676 6 2024-04-20T02:50:09.673 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:09.673+0000 7f0bfac6c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-20T02:50:09.730 INFO:teuthology.orchestra.run.smithi138.stdout:6 2024-04-20T02:50:09.744 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new 53ae8193-5c3e-4698-aacd-df2ad1db9143 7 2024-04-20T02:50:09.769 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:09.770+0000 7f609f66b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-20T02:50:09.958 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:09.959+0000 7f609f66b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-20T02:50:09.989 INFO:teuthology.orchestra.run.smithi138.stdout:7 2024-04-20T02:50:09.992 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:09.993+0000 7f0bfac6c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-20T02:50:10.002 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-20T02:50:10.002 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-20T02:50:10.004 INFO:tasks.ceph.osd.0:Started 2024-04-20T02:50:10.004 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-20T02:50:10.004 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-20T02:50:10.006 INFO:tasks.ceph.osd.1:Started 2024-04-20T02:50:10.006 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-20T02:50:10.006 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-20T02:50:10.009 INFO:tasks.ceph.osd.2:Started 2024-04-20T02:50:10.009 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-20T02:50:10.009 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-20T02:50:10.011 INFO:tasks.ceph.osd.3:Started 2024-04-20T02:50:10.011 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-20T02:50:10.011 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-20T02:50:10.041 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:10.041+0000 7f609f66b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-20T02:50:10.042 INFO:tasks.ceph.osd.4:Started 2024-04-20T02:50:10.043 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-20T02:50:10.043 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-20T02:50:10.045 INFO:tasks.ceph.osd.5:Started 2024-04-20T02:50:10.045 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-20T02:50:10.045 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-20T02:50:10.047 INFO:tasks.ceph.osd.6:Started 2024-04-20T02:50:10.048 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-20T02:50:10.048 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-20T02:50:10.050 INFO:tasks.ceph.osd.7:Started 2024-04-20T02:50:10.050 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T02:50:10.199 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:10.199+0000 7f0bfac6c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-20T02:50:10.266 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:50:10.266 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":10,"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","created":"2024-04-20T02:50:07.356902+0000","modified":"2024-04-20T02:50:09.985734+0000","last_up_change":"0.000000","last_in_change":"2024-04-20T02:50:09.985734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ae8a64-fc7c-47ab-af6a-4b762a840645","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"336e852e-5fa6-4ac0-b053-d31ad40d7c4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"695cab3a-74d8-4046-ac61-2656dc0d893e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d41da44-d894-40ac-acaa-aa0df15296c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3e1c33fa-6c8c-4898-99bc-462ecef25f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd1fc124-7cf8-4628-9076-1f27d6385676","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"53ae8193-5c3e-4698-aacd-df2ad1db9143","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T02:50:10.278 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-20T02:50:10.279 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-20T02:50:10.280 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:10.280+0000 7f0bfac6c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-20T02:50:10.612 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:10.612+0000 7f34f3ab17c0 -1 Falling back to public interface 2024-04-20T02:50:10.632 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:10.632+0000 7ff2c0d377c0 -1 Falling back to public interface 2024-04-20T02:50:10.633 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:10.633+0000 7f276f22a7c0 -1 Falling back to public interface 2024-04-20T02:50:10.636 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:10.636+0000 7f07bbfd97c0 -1 Falling back to public interface 2024-04-20T02:50:10.673 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:10.673+0000 7f80f32077c0 -1 Falling back to public interface 2024-04-20T02:50:10.673 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:10.673+0000 7f78e4c277c0 -1 Falling back to public interface 2024-04-20T02:50:10.674 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:10.673+0000 7f80ea5c5700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-04-20T02:50:10.674 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:10.673+0000 7f80ea5c5700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2024-04-20T02:50:10.676 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:10.677+0000 7f6a7a96c7c0 -1 Falling back to public interface 2024-04-20T02:50:10.681 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:10.681+0000 7fdd453877c0 -1 Falling back to public interface 2024-04-20T02:50:10.764 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:10.764+0000 7f609f66b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-20T02:50:10.846 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:10.846+0000 7f609f66b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-20T02:50:10.979 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:10.979+0000 7f609f66b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-20T02:50:10.990 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:10.990+0000 7f0bfac6c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-20T02:50:11.072 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:11.072+0000 7f0bfac6c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-20T02:50:11.085 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-20T02:50:11.211 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:11.211+0000 7f0bfac6c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-20T02:50:11.279 INFO:teuthology.misc.health.smithi105.stdout: 2024-04-20T02:50:11.279 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":10,"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","created":"2024-04-20T02:50:07.356902+0000","modified":"2024-04-20T02:50:09.985734+0000","last_up_change":"0.000000","last_in_change":"2024-04-20T02:50:09.985734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ae8a64-fc7c-47ab-af6a-4b762a840645","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"336e852e-5fa6-4ac0-b053-d31ad40d7c4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"695cab3a-74d8-4046-ac61-2656dc0d893e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d41da44-d894-40ac-acaa-aa0df15296c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3e1c33fa-6c8c-4898-99bc-462ecef25f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd1fc124-7cf8-4628-9076-1f27d6385676","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"53ae8193-5c3e-4698-aacd-df2ad1db9143","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T02:50:11.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-20T02:50:11.528 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:11.529+0000 7f609f66b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-20T02:50:11.699 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:11.700+0000 7f609f66b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-20T02:50:11.762 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:11.762+0000 7f0bfac6c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-20T02:50:11.866 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:11.866+0000 7f609f66b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-20T02:50:11.930 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:11.930+0000 7f0bfac6c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-20T02:50:12.098 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:12.098+0000 7f0bfac6c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-20T02:50:12.701 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:12.701+0000 7f609f66b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-20T02:50:12.924 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:12.925+0000 7f0bfac6c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-20T02:50:12.938 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:12.939+0000 7f609f66b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-20T02:50:13.013 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:13.013+0000 7f609f66b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-20T02:50:13.170 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:13.170+0000 7f0bfac6c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-20T02:50:13.243 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:13.243+0000 7f0bfac6c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-20T02:50:13.412 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:13.412+0000 7f609f66b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-20T02:50:13.684 INFO:tasks.ceph.mgr.y.smithi105.stderr:2024-04-20T02:50:13.684+0000 7f0bfac6c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-20T02:50:14.109 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:14.110+0000 7f34f3ab17c0 -1 osd.2 0 log_to_monitors true 2024-04-20T02:50:14.142 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:14.142+0000 7f07bbfd97c0 -1 osd.0 0 log_to_monitors true 2024-04-20T02:50:14.152 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:14.153+0000 7f276f22a7c0 -1 osd.3 0 log_to_monitors true 2024-04-20T02:50:14.159 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:14.160+0000 7ff2c0d377c0 -1 osd.1 0 log_to_monitors true 2024-04-20T02:50:14.194 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:14.194+0000 7f6a7a96c7c0 -1 osd.4 0 log_to_monitors true 2024-04-20T02:50:14.209 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:14.210+0000 7f78e4c277c0 -1 osd.6 0 log_to_monitors true 2024-04-20T02:50:14.213 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:14.214+0000 7f80f32077c0 -1 osd.5 0 log_to_monitors true 2024-04-20T02:50:14.219 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:14.220+0000 7fdd453877c0 -1 osd.7 0 log_to_monitors true 2024-04-20T02:50:14.441 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:14.441+0000 7f605a8e1700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-20T02:50:14.441 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:50:14.442+0000 7f605a8e1700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-20T02:50:15.557 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:15.557+0000 7f78d8fdf700 -1 osd.6 0 waiting for initial osdmap 2024-04-20T02:50:15.558 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:15.558+0000 7f27635e2700 -1 osd.3 0 waiting for initial osdmap 2024-04-20T02:50:15.565 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:15.566+0000 7f275f1c4700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:50:15.567 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:15.568+0000 7f78d4bc1700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:50:15.569 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:15.570+0000 7ff2b50ef700 -1 osd.1 0 waiting for initial osdmap 2024-04-20T02:50:15.570 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:15.570+0000 7f34e7e69700 -1 osd.2 0 waiting for initial osdmap 2024-04-20T02:50:15.573 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:15.573+0000 7f34e3a4b700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:50:15.574 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:15.574+0000 7ff2b0cd1700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:50:18.098 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-20T02:50:18.256 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:18.257+0000 7f07b0391700 -1 osd.0 0 waiting for initial osdmap 2024-04-20T02:50:18.259 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:18.260+0000 7f07a7f6b700 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:50:18.449 INFO:teuthology.misc.health.smithi105.stdout: 2024-04-20T02:50:18.449 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":14,"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","created":"2024-04-20T02:50:07.356902+0000","modified":"2024-04-20T02:50:17.569030+0000","last_up_change":"2024-04-20T02:50:16.560939+0000","last_in_change":"2024-04-20T02:50:09.985734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T02:50:17.498569+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c5ae8a64-fc7c-47ab-af6a-4b762a840645","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6805","nonce":43331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6807","nonce":43331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43331}]},"public_addr":"172.21.15.105:6804/43331","cluster_addr":"172.21.15.105:6805/43331","heartbeat_back_addr":"172.21.15.105:6807/43331","heartbeat_front_addr":"172.21.15.105:6806/43331","state":["exists","up"]},{"osd":2,"uuid":"336e852e-5fa6-4ac0-b053-d31ad40d7c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":43326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6801","nonce":43326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6803","nonce":43326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":43326}]},"public_addr":"172.21.15.105:6800/43326","cluster_addr":"172.21.15.105:6801/43326","heartbeat_back_addr":"172.21.15.105:6803/43326","heartbeat_front_addr":"172.21.15.105:6802/43326","state":["exists","up"]},{"osd":3,"uuid":"695cab3a-74d8-4046-ac61-2656dc0d893e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6809","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6811","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":43329}]},"public_addr":"172.21.15.105:6808/43329","cluster_addr":"172.21.15.105:6809/43329","heartbeat_back_addr":"172.21.15.105:6811/43329","heartbeat_front_addr":"172.21.15.105:6810/43329","state":["exists","up"]},{"osd":4,"uuid":"1d41da44-d894-40ac-acaa-aa0df15296c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3e1c33fa-6c8c-4898-99bc-462ecef25f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd1fc124-7cf8-4628-9076-1f27d6385676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":43329}]},"public_addr":"172.21.15.138:6800/43329","cluster_addr":"172.21.15.138:6801/43329","heartbeat_back_addr":"172.21.15.138:6805/43329","heartbeat_front_addr":"172.21.15.138:6803/43329","state":["exists","up"]},{"osd":7,"uuid":"53ae8193-5c3e-4698-aacd-df2ad1db9143","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.118746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.160844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.195185+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.247500+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T02:50:18.463 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-04-20T02:50:18.480 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:18.481+0000 7f80e75bf700 -1 osd.5 0 waiting for initial osdmap 2024-04-20T02:50:18.485 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:18.486+0000 7f80df199700 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:50:18.490 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:18.490+0000 7fdd3973f700 -1 osd.7 0 waiting for initial osdmap 2024-04-20T02:50:18.493 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:18.493+0000 7fdd31319700 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:50:18.511 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:18.511+0000 7f6a6ed24700 -1 osd.4 0 waiting for initial osdmap 2024-04-20T02:50:18.514 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:18.514+0000 7f6a668fe700 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:50:25.272 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-20T02:50:25.602 INFO:teuthology.misc.health.smithi105.stdout: 2024-04-20T02:50:25.602 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":17,"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","created":"2024-04-20T02:50:07.356902+0000","modified":"2024-04-20T02:50:20.594633+0000","last_up_change":"2024-04-20T02:50:18.574866+0000","last_in_change":"2024-04-20T02:50:09.985734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T02:50:17.498569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":43330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":43330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6815","nonce":43330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":43330}]},"public_addr":"172.21.15.105:6812/43330","cluster_addr":"172.21.15.105:6813/43330","heartbeat_back_addr":"172.21.15.105:6815/43330","heartbeat_front_addr":"172.21.15.105:6814/43330","state":["exists","up"]},{"osd":1,"uuid":"c5ae8a64-fc7c-47ab-af6a-4b762a840645","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6805","nonce":43331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6807","nonce":43331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43331}]},"public_addr":"172.21.15.105:6804/43331","cluster_addr":"172.21.15.105:6805/43331","heartbeat_back_addr":"172.21.15.105:6807/43331","heartbeat_front_addr":"172.21.15.105:6806/43331","state":["exists","up"]},{"osd":2,"uuid":"336e852e-5fa6-4ac0-b053-d31ad40d7c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":43326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6801","nonce":43326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6803","nonce":43326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":43326}]},"public_addr":"172.21.15.105:6800/43326","cluster_addr":"172.21.15.105:6801/43326","heartbeat_back_addr":"172.21.15.105:6803/43326","heartbeat_front_addr":"172.21.15.105:6802/43326","state":["exists","up"]},{"osd":3,"uuid":"695cab3a-74d8-4046-ac61-2656dc0d893e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6809","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6811","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":43329}]},"public_addr":"172.21.15.105:6808/43329","cluster_addr":"172.21.15.105:6809/43329","heartbeat_back_addr":"172.21.15.105:6811/43329","heartbeat_front_addr":"172.21.15.105:6810/43329","state":["exists","up"]},{"osd":4,"uuid":"1d41da44-d894-40ac-acaa-aa0df15296c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":43330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":43330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":43330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":43330}]},"public_addr":"172.21.15.138:6808/43330","cluster_addr":"172.21.15.138:6809/43330","heartbeat_back_addr":"172.21.15.138:6811/43330","heartbeat_front_addr":"172.21.15.138:6810/43330","state":["exists","up"]},{"osd":5,"uuid":"3e1c33fa-6c8c-4898-99bc-462ecef25f30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":43328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":43328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":43328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":43328}]},"public_addr":"172.21.15.138:6802/43328","cluster_addr":"172.21.15.138:6804/43328","heartbeat_back_addr":"172.21.15.138:6807/43328","heartbeat_front_addr":"172.21.15.138:6806/43328","state":["exists","up"]},{"osd":6,"uuid":"fd1fc124-7cf8-4628-9076-1f27d6385676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":43329}]},"public_addr":"172.21.15.138:6800/43329","cluster_addr":"172.21.15.138:6801/43329","heartbeat_back_addr":"172.21.15.138:6805/43329","heartbeat_front_addr":"172.21.15.138:6803/43329","state":["exists","up"]},{"osd":7,"uuid":"53ae8193-5c3e-4698-aacd-df2ad1db9143","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":43351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6813","nonce":43351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6815","nonce":43351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":43351}]},"public_addr":"172.21.15.138:6812/43351","cluster_addr":"172.21.15.138:6813/43351","heartbeat_back_addr":"172.21.15.138:6815/43351","heartbeat_front_addr":"172.21.15.138:6814/43351","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.140763+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.118746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.160844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.195185+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.241192+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.255506+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.247500+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.184132+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T02:50:25.615 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-20T02:50:25.616 INFO:tasks.ceph:Creating RBD pool 2024-04-20T02:50:25.616 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-20T02:50:26.633 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'rbd' created 2024-04-20T02:50:26.647 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-20T02:50:27.638 INFO:teuthology.orchestra.run.smithi105.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-20T02:50:27.650 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-20T02:50:27.651 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-20T02:50:27.651 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-20T02:50:27.651 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-20T02:50:27.651 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-20T02:50:28.041 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:50:28.053 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":42698}]},"active_addr":"172.21.15.138:6816/42698","active_change":"2024-04-20T02:50:13.428316+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4106,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":1859404371}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":635542909}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":637991798}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":3565510621}]}]} 2024-04-20T02:50:28.059 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-20T02:50:28.059 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-20T02:50:28.060 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T02:50:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:50:28.386 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":19,"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","created":"2024-04-20T02:50:07.356902+0000","modified":"2024-04-20T02:50:27.625284+0000","last_up_change":"2024-04-20T02:50:18.574866+0000","last_in_change":"2024-04-20T02:50:09.985734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T02:50:17.498569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-20T02:50:25.955448+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":43330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":43330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6815","nonce":43330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":43330}]},"public_addr":"172.21.15.105:6812/43330","cluster_addr":"172.21.15.105:6813/43330","heartbeat_back_addr":"172.21.15.105:6815/43330","heartbeat_front_addr":"172.21.15.105:6814/43330","state":["exists","up"]},{"osd":1,"uuid":"c5ae8a64-fc7c-47ab-af6a-4b762a840645","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6805","nonce":43331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6807","nonce":43331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43331}]},"public_addr":"172.21.15.105:6804/43331","cluster_addr":"172.21.15.105:6805/43331","heartbeat_back_addr":"172.21.15.105:6807/43331","heartbeat_front_addr":"172.21.15.105:6806/43331","state":["exists","up"]},{"osd":2,"uuid":"336e852e-5fa6-4ac0-b053-d31ad40d7c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":43326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6801","nonce":43326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6803","nonce":43326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":43326}]},"public_addr":"172.21.15.105:6800/43326","cluster_addr":"172.21.15.105:6801/43326","heartbeat_back_addr":"172.21.15.105:6803/43326","heartbeat_front_addr":"172.21.15.105:6802/43326","state":["exists","up"]},{"osd":3,"uuid":"695cab3a-74d8-4046-ac61-2656dc0d893e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6809","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6811","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":43329}]},"public_addr":"172.21.15.105:6808/43329","cluster_addr":"172.21.15.105:6809/43329","heartbeat_back_addr":"172.21.15.105:6811/43329","heartbeat_front_addr":"172.21.15.105:6810/43329","state":["exists","up"]},{"osd":4,"uuid":"1d41da44-d894-40ac-acaa-aa0df15296c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":43330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":43330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":43330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":43330}]},"public_addr":"172.21.15.138:6808/43330","cluster_addr":"172.21.15.138:6809/43330","heartbeat_back_addr":"172.21.15.138:6811/43330","heartbeat_front_addr":"172.21.15.138:6810/43330","state":["exists","up"]},{"osd":5,"uuid":"3e1c33fa-6c8c-4898-99bc-462ecef25f30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":43328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":43328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":43328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":43328}]},"public_addr":"172.21.15.138:6802/43328","cluster_addr":"172.21.15.138:6804/43328","heartbeat_back_addr":"172.21.15.138:6807/43328","heartbeat_front_addr":"172.21.15.138:6806/43328","state":["exists","up"]},{"osd":6,"uuid":"fd1fc124-7cf8-4628-9076-1f27d6385676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":43329}]},"public_addr":"172.21.15.138:6800/43329","cluster_addr":"172.21.15.138:6801/43329","heartbeat_back_addr":"172.21.15.138:6805/43329","heartbeat_front_addr":"172.21.15.138:6803/43329","state":["exists","up"]},{"osd":7,"uuid":"53ae8193-5c3e-4698-aacd-df2ad1db9143","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":43351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6813","nonce":43351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6815","nonce":43351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":43351}]},"public_addr":"172.21.15.138:6812/43351","cluster_addr":"172.21.15.138:6813/43351","heartbeat_back_addr":"172.21.15.138:6815/43351","heartbeat_front_addr":"172.21.15.138:6814/43351","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.140763+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.118746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.160844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.195185+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.241192+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.255506+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.247500+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.184132+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T02:50:28.400 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-20T02:50:28.400 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T02:50:28.732 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:50:28.732 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":20,"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","created":"2024-04-20T02:50:07.356902+0000","modified":"2024-04-20T02:50:28.629861+0000","last_up_change":"2024-04-20T02:50:18.574866+0000","last_in_change":"2024-04-20T02:50:09.985734+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T02:50:17.498569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-20T02:50:25.955448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":43330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":43330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6815","nonce":43330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":43330}]},"public_addr":"172.21.15.105:6812/43330","cluster_addr":"172.21.15.105:6813/43330","heartbeat_back_addr":"172.21.15.105:6815/43330","heartbeat_front_addr":"172.21.15.105:6814/43330","state":["exists","up"]},{"osd":1,"uuid":"c5ae8a64-fc7c-47ab-af6a-4b762a840645","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6805","nonce":43331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6807","nonce":43331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43331}]},"public_addr":"172.21.15.105:6804/43331","cluster_addr":"172.21.15.105:6805/43331","heartbeat_back_addr":"172.21.15.105:6807/43331","heartbeat_front_addr":"172.21.15.105:6806/43331","state":["exists","up"]},{"osd":2,"uuid":"336e852e-5fa6-4ac0-b053-d31ad40d7c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":43326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6801","nonce":43326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6803","nonce":43326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":43326}]},"public_addr":"172.21.15.105:6800/43326","cluster_addr":"172.21.15.105:6801/43326","heartbeat_back_addr":"172.21.15.105:6803/43326","heartbeat_front_addr":"172.21.15.105:6802/43326","state":["exists","up"]},{"osd":3,"uuid":"695cab3a-74d8-4046-ac61-2656dc0d893e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6809","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6811","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":43329}]},"public_addr":"172.21.15.105:6808/43329","cluster_addr":"172.21.15.105:6809/43329","heartbeat_back_addr":"172.21.15.105:6811/43329","heartbeat_front_addr":"172.21.15.105:6810/43329","state":["exists","up"]},{"osd":4,"uuid":"1d41da44-d894-40ac-acaa-aa0df15296c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":43330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":43330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":43330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":43330}]},"public_addr":"172.21.15.138:6808/43330","cluster_addr":"172.21.15.138:6809/43330","heartbeat_back_addr":"172.21.15.138:6811/43330","heartbeat_front_addr":"172.21.15.138:6810/43330","state":["exists","up"]},{"osd":5,"uuid":"3e1c33fa-6c8c-4898-99bc-462ecef25f30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":43328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":43328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":43328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":43328}]},"public_addr":"172.21.15.138:6802/43328","cluster_addr":"172.21.15.138:6804/43328","heartbeat_back_addr":"172.21.15.138:6807/43328","heartbeat_front_addr":"172.21.15.138:6806/43328","state":["exists","up"]},{"osd":6,"uuid":"fd1fc124-7cf8-4628-9076-1f27d6385676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":43329}]},"public_addr":"172.21.15.138:6800/43329","cluster_addr":"172.21.15.138:6801/43329","heartbeat_back_addr":"172.21.15.138:6805/43329","heartbeat_front_addr":"172.21.15.138:6803/43329","state":["exists","up"]},{"osd":7,"uuid":"53ae8193-5c3e-4698-aacd-df2ad1db9143","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":43351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6813","nonce":43351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6815","nonce":43351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":43351}]},"public_addr":"172.21.15.138:6812/43351","cluster_addr":"172.21.15.138:6813/43351","heartbeat_back_addr":"172.21.15.138:6815/43351","heartbeat_front_addr":"172.21.15.138:6814/43351","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.140763+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.118746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.160844+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.195185+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.241192+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.255506+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.247500+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:50:15.184132+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T02:50:28.747 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-20T02:50:28.748 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-20T02:50:28.748 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-20T02:50:28.749 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-20T02:50:28.749 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-20T02:50:28.750 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-20T02:50:28.750 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-20T02:50:28.751 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-20T02:50:29.031 INFO:teuthology.orchestra.run.smithi105.stdout:55834574852 2024-04-20T02:50:29.031 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-20T02:50:29.039 INFO:teuthology.orchestra.run.smithi105.stdout:55834574853 2024-04-20T02:50:29.039 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-20T02:50:29.047 INFO:teuthology.orchestra.run.smithi105.stdout:55834574854 2024-04-20T02:50:29.048 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-20T02:50:29.052 INFO:teuthology.orchestra.run.smithi105.stdout:64424509443 2024-04-20T02:50:29.053 INFO:teuthology.orchestra.run.smithi105.stdout:55834574852 2024-04-20T02:50:29.053 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-20T02:50:29.054 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-20T02:50:29.055 INFO:teuthology.orchestra.run.smithi105.stdout:64424509445 2024-04-20T02:50:29.055 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-20T02:50:29.063 INFO:teuthology.orchestra.run.smithi105.stdout:64424509447 2024-04-20T02:50:29.063 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-20T02:50:29.066 INFO:teuthology.orchestra.run.smithi105.stdout:64424509444 2024-04-20T02:50:29.066 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-20T02:50:29.696 INFO:teuthology.orchestra.run.smithi105.stdout:55834574853 2024-04-20T02:50:29.700 INFO:teuthology.orchestra.run.smithi105.stdout:64424509444 2024-04-20T02:50:29.706 INFO:teuthology.orchestra.run.smithi105.stdout:55834574852 2024-04-20T02:50:29.725 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.5 2024-04-20T02:50:29.725 DEBUG:teuthology.parallel:result is None 2024-04-20T02:50:29.728 INFO:teuthology.orchestra.run.smithi105.stdout:64424509447 2024-04-20T02:50:29.730 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.1 2024-04-20T02:50:29.730 DEBUG:teuthology.parallel:result is None 2024-04-20T02:50:29.731 INFO:teuthology.orchestra.run.smithi105.stdout:55834574852 2024-04-20T02:50:29.732 INFO:teuthology.orchestra.run.smithi105.stdout:64424509443 2024-04-20T02:50:29.739 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2024-04-20T02:50:29.739 INFO:teuthology.orchestra.run.smithi105.stdout:64424509445 2024-04-20T02:50:29.739 DEBUG:teuthology.parallel:result is None 2024-04-20T02:50:29.748 INFO:teuthology.orchestra.run.smithi105.stdout:55834574854 2024-04-20T02:50:29.753 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2024-04-20T02:50:29.753 DEBUG:teuthology.parallel:result is None 2024-04-20T02:50:29.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.0 2024-04-20T02:50:29.754 DEBUG:teuthology.parallel:result is None 2024-04-20T02:50:29.758 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.7 2024-04-20T02:50:29.758 DEBUG:teuthology.parallel:result is None 2024-04-20T02:50:29.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.4 2024-04-20T02:50:29.762 DEBUG:teuthology.parallel:result is None 2024-04-20T02:50:29.765 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-04-20T02:50:29.765 DEBUG:teuthology.parallel:result is None 2024-04-20T02:50:29.765 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-20T02:50:29.765 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:50:30.129 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:50:30.131 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:50:30.143 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-20T02:50:29.440084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217620,"kb_used_data":1436,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757100,"statfs":{"total":773094113280,"available":772871270400,"internally_reserved":0,"allocated":1470464,"data_stored":2004864,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000916"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:27.643272+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:50:27.643272+0000","last_peered":"2024-04-20T02:50:27.643272+0000","last_clean":"2024-04-20T02:50:27.643272+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:50:27.643272+0000","last_undegraded":"2024-04-20T02:50:27.643272+0000","last_fullsized":"2024-04-20T02:50:27.643272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.432280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:27.638986+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:50:27.638986+0000","last_peered":"2024-04-20T02:50:27.638986+0000","last_clean":"2024-04-20T02:50:27.638986+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:50:27.638986+0000","last_undegraded":"2024-04-20T02:50:27.638986+0000","last_fullsized":"2024-04-20T02:50:27.638986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.092656+0000","last_change":"2024-04-20T02:50:28.091938+0000","last_active":"2024-04-20T02:50:28.092656+0000","last_peered":"2024-04-20T02:50:28.092656+0000","last_clean":"2024-04-20T02:50:28.092656+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:50:28.092656+0000","last_undegraded":"2024-04-20T02:50:28.092656+0000","last_fullsized":"2024-04-20T02:50:28.092656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:34.990718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.092908+0000","last_change":"2024-04-20T02:50:28.092115+0000","last_active":"2024-04-20T02:50:28.092908+0000","last_peered":"2024-04-20T02:50:28.092908+0000","last_clean":"2024-04-20T02:50:28.092908+0000","last_became_active":"2024-04-20T02:50:28.091570+0000","last_became_peered":"2024-04-20T02:50:28.091570+0000","last_unstale":"2024-04-20T02:50:28.092908+0000","last_undegraded":"2024-04-20T02:50:28.092908+0000","last_fullsized":"2024-04-20T02:50:28.092908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:27.403210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-20T02:50:26.631045+0000","last_change":"2024-04-20T02:50:19.923431+0000","last_active":"2024-04-20T02:50:26.631045+0000","last_peered":"2024-04-20T02:50:26.631045+0000","last_clean":"2024-04-20T02:50:26.631045+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:50:26.631045+0000","last_undegraded":"2024-04-20T02:50:26.631045+0000","last_fullsized":"2024-04-20T02:50:26.631045+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:50:17.569030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.655607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.639459+0000","last_change":"2024-04-20T02:50:27.638361+0000","last_active":"2024-04-20T02:50:28.639459+0000","last_peered":"2024-04-20T02:50:28.639459+0000","last_clean":"2024-04-20T02:50:28.639459+0000","last_became_active":"2024-04-20T02:50:27.637907+0000","last_became_peered":"2024-04-20T02:50:27.637907+0000","last_unstale":"2024-04-20T02:50:28.639459+0000","last_undegraded":"2024-04-20T02:50:28.639459+0000","last_fullsized":"2024-04-20T02:50:28.639459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:41.417237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.091742+0000","last_change":"2024-04-20T02:50:28.091742+0000","last_active":"2024-04-20T02:50:28.091742+0000","last_peered":"2024-04-20T02:50:28.091742+0000","last_clean":"2024-04-20T02:50:28.091742+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:50:28.091742+0000","last_undegraded":"2024-04-20T02:50:28.091742+0000","last_fullsized":"2024-04-20T02:50:28.091742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:32.326186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.639461+0000","last_change":"2024-04-20T02:50:27.638420+0000","last_active":"2024-04-20T02:50:28.639461+0000","last_peered":"2024-04-20T02:50:28.639461+0000","last_clean":"2024-04-20T02:50:28.639461+0000","last_became_active":"2024-04-20T02:50:27.638143+0000","last_became_peered":"2024-04-20T02:50:27.638143+0000","last_unstale":"2024-04-20T02:50:28.639461+0000","last_undegraded":"2024-04-20T02:50:28.639461+0000","last_fullsized":"2024-04-20T02:50:28.639461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.959545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.091519+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:50:28.091519+0000","last_peered":"2024-04-20T02:50:28.091519+0000","last_clean":"2024-04-20T02:50:28.091519+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:50:28.091519+0000","last_undegraded":"2024-04-20T02:50:28.091519+0000","last_fullsized":"2024-04-20T02:50:28.091519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":581193,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]}]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":56893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":581193,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":576525,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:50:30.144 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:50:30.469 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:50:30.472 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:50:30.483 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-20T02:50:29.440084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217620,"kb_used_data":1436,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757100,"statfs":{"total":773094113280,"available":772871270400,"internally_reserved":0,"allocated":1470464,"data_stored":2004864,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000916"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:27.643272+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:50:27.643272+0000","last_peered":"2024-04-20T02:50:27.643272+0000","last_clean":"2024-04-20T02:50:27.643272+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:50:27.643272+0000","last_undegraded":"2024-04-20T02:50:27.643272+0000","last_fullsized":"2024-04-20T02:50:27.643272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.432280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:27.638986+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:50:27.638986+0000","last_peered":"2024-04-20T02:50:27.638986+0000","last_clean":"2024-04-20T02:50:27.638986+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:50:27.638986+0000","last_undegraded":"2024-04-20T02:50:27.638986+0000","last_fullsized":"2024-04-20T02:50:27.638986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.092656+0000","last_change":"2024-04-20T02:50:28.091938+0000","last_active":"2024-04-20T02:50:28.092656+0000","last_peered":"2024-04-20T02:50:28.092656+0000","last_clean":"2024-04-20T02:50:28.092656+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:50:28.092656+0000","last_undegraded":"2024-04-20T02:50:28.092656+0000","last_fullsized":"2024-04-20T02:50:28.092656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:34.990718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.092908+0000","last_change":"2024-04-20T02:50:28.092115+0000","last_active":"2024-04-20T02:50:28.092908+0000","last_peered":"2024-04-20T02:50:28.092908+0000","last_clean":"2024-04-20T02:50:28.092908+0000","last_became_active":"2024-04-20T02:50:28.091570+0000","last_became_peered":"2024-04-20T02:50:28.091570+0000","last_unstale":"2024-04-20T02:50:28.092908+0000","last_undegraded":"2024-04-20T02:50:28.092908+0000","last_fullsized":"2024-04-20T02:50:28.092908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:27.403210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-04-20T02:50:26.631045+0000","last_change":"2024-04-20T02:50:19.923431+0000","last_active":"2024-04-20T02:50:26.631045+0000","last_peered":"2024-04-20T02:50:26.631045+0000","last_clean":"2024-04-20T02:50:26.631045+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:50:26.631045+0000","last_undegraded":"2024-04-20T02:50:26.631045+0000","last_fullsized":"2024-04-20T02:50:26.631045+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:50:17.569030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.655607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.639459+0000","last_change":"2024-04-20T02:50:27.638361+0000","last_active":"2024-04-20T02:50:28.639459+0000","last_peered":"2024-04-20T02:50:28.639459+0000","last_clean":"2024-04-20T02:50:28.639459+0000","last_became_active":"2024-04-20T02:50:27.637907+0000","last_became_peered":"2024-04-20T02:50:27.637907+0000","last_unstale":"2024-04-20T02:50:28.639459+0000","last_undegraded":"2024-04-20T02:50:28.639459+0000","last_fullsized":"2024-04-20T02:50:28.639459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:41.417237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.091742+0000","last_change":"2024-04-20T02:50:28.091742+0000","last_active":"2024-04-20T02:50:28.091742+0000","last_peered":"2024-04-20T02:50:28.091742+0000","last_clean":"2024-04-20T02:50:28.091742+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:50:28.091742+0000","last_undegraded":"2024-04-20T02:50:28.091742+0000","last_fullsized":"2024-04-20T02:50:28.091742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:32.326186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.639461+0000","last_change":"2024-04-20T02:50:27.638420+0000","last_active":"2024-04-20T02:50:28.639461+0000","last_peered":"2024-04-20T02:50:28.639461+0000","last_clean":"2024-04-20T02:50:28.639461+0000","last_became_active":"2024-04-20T02:50:27.638143+0000","last_became_peered":"2024-04-20T02:50:27.638143+0000","last_unstale":"2024-04-20T02:50:28.639461+0000","last_undegraded":"2024-04-20T02:50:28.639461+0000","last_fullsized":"2024-04-20T02:50:28.639461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.959545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.091519+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:50:28.091519+0000","last_peered":"2024-04-20T02:50:28.091519+0000","last_clean":"2024-04-20T02:50:28.091519+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:50:28.091519+0000","last_undegraded":"2024-04-20T02:50:28.091519+0000","last_fullsized":"2024-04-20T02:50:28.091519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":581193,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]}]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":56893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":581193,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":576525,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:50:30.484 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-20T02:50:30.484 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-20T02:50:30.484 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-20T02:50:30.484 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-20T02:50:30.854 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:50:30.854 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-20T02:50:30.868 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-20T02:50:30.868 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-20T02:50:30.881 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-20T02:50:30.881 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri7-testing-2024-04-18-1351-reef'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'min_in': 4}, 'workunit': {'branch': 'wip-yuri7-testing-2024-04-18-1351-reef', 'sha1': '15eecde2bab9803889d929c7e423d07970d8cbed'}} 2024-04-20T02:50:30.881 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'min_in': 4} 2024-04-20T02:50:30.881 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-20T02:50:31.228 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-20T02:50:31.228 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-20T02:50:31.229 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:50:31.555 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:50:31.557 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:50:31.569 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-04-20T02:50:31.440631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217656,"kb_used_data":1468,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757064,"statfs":{"total":773094113280,"available":772871233536,"internally_reserved":0,"allocated":1503232,"data_stored":2023416,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001463"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:27.643272+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:50:27.643272+0000","last_peered":"2024-04-20T02:50:27.643272+0000","last_clean":"2024-04-20T02:50:27.643272+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:50:27.643272+0000","last_undegraded":"2024-04-20T02:50:27.643272+0000","last_fullsized":"2024-04-20T02:50:27.643272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.432280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:27.638986+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:50:27.638986+0000","last_peered":"2024-04-20T02:50:27.638986+0000","last_clean":"2024-04-20T02:50:27.638986+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:50:27.638986+0000","last_undegraded":"2024-04-20T02:50:27.638986+0000","last_fullsized":"2024-04-20T02:50:27.638986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:29.030331+0000","last_change":"2024-04-20T02:50:28.091938+0000","last_active":"2024-04-20T02:50:29.030331+0000","last_peered":"2024-04-20T02:50:29.030331+0000","last_clean":"2024-04-20T02:50:29.030331+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:50:29.030331+0000","last_undegraded":"2024-04-20T02:50:29.030331+0000","last_fullsized":"2024-04-20T02:50:29.030331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:34.990718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:29.030307+0000","last_change":"2024-04-20T02:50:28.092115+0000","last_active":"2024-04-20T02:50:29.030307+0000","last_peered":"2024-04-20T02:50:29.030307+0000","last_clean":"2024-04-20T02:50:29.030307+0000","last_became_active":"2024-04-20T02:50:28.091570+0000","last_became_peered":"2024-04-20T02:50:28.091570+0000","last_unstale":"2024-04-20T02:50:29.030307+0000","last_undegraded":"2024-04-20T02:50:29.030307+0000","last_fullsized":"2024-04-20T02:50:29.030307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:27.403210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:27.636042+0000","last_change":"2024-04-20T02:50:19.923431+0000","last_active":"2024-04-20T02:50:27.636042+0000","last_peered":"2024-04-20T02:50:27.636042+0000","last_clean":"2024-04-20T02:50:27.636042+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:50:27.636042+0000","last_undegraded":"2024-04-20T02:50:27.636042+0000","last_fullsized":"2024-04-20T02:50:27.636042+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:50:17.569030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.655607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.639459+0000","last_change":"2024-04-20T02:50:27.638361+0000","last_active":"2024-04-20T02:50:28.639459+0000","last_peered":"2024-04-20T02:50:28.639459+0000","last_clean":"2024-04-20T02:50:28.639459+0000","last_became_active":"2024-04-20T02:50:27.637907+0000","last_became_peered":"2024-04-20T02:50:27.637907+0000","last_unstale":"2024-04-20T02:50:28.639459+0000","last_undegraded":"2024-04-20T02:50:28.639459+0000","last_fullsized":"2024-04-20T02:50:28.639459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:41.417237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.091742+0000","last_change":"2024-04-20T02:50:28.091742+0000","last_active":"2024-04-20T02:50:28.091742+0000","last_peered":"2024-04-20T02:50:28.091742+0000","last_clean":"2024-04-20T02:50:28.091742+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:50:28.091742+0000","last_undegraded":"2024-04-20T02:50:28.091742+0000","last_fullsized":"2024-04-20T02:50:28.091742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:32.326186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.639461+0000","last_change":"2024-04-20T02:50:27.638420+0000","last_active":"2024-04-20T02:50:28.639461+0000","last_peered":"2024-04-20T02:50:28.639461+0000","last_clean":"2024-04-20T02:50:28.639461+0000","last_became_active":"2024-04-20T02:50:27.638143+0000","last_became_peered":"2024-04-20T02:50:27.638143+0000","last_unstale":"2024-04-20T02:50:28.639461+0000","last_undegraded":"2024-04-20T02:50:28.639461+0000","last_fullsized":"2024-04-20T02:50:28.639461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.959545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.091519+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:50:28.091519+0000","last_peered":"2024-04-20T02:50:28.091519+0000","last_clean":"2024-04-20T02:50:28.091519+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:50:28.091519+0000","last_undegraded":"2024-04-20T02:50:28.091519+0000","last_fullsized":"2024-04-20T02:50:28.091519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509448,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":585821,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]}]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":56893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":581193,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":585821,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":172032,"data_stored":56893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:50:31.570 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:50:31.897 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:50:31.899 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:50:31.911 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-04-20T02:50:31.440631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217656,"kb_used_data":1468,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757064,"statfs":{"total":773094113280,"available":772871233536,"internally_reserved":0,"allocated":1503232,"data_stored":2023416,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001463"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:27.643272+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:50:27.643272+0000","last_peered":"2024-04-20T02:50:27.643272+0000","last_clean":"2024-04-20T02:50:27.643272+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:50:27.643272+0000","last_undegraded":"2024-04-20T02:50:27.643272+0000","last_fullsized":"2024-04-20T02:50:27.643272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.432280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:27.638986+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:50:27.638986+0000","last_peered":"2024-04-20T02:50:27.638986+0000","last_clean":"2024-04-20T02:50:27.638986+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:50:27.638986+0000","last_undegraded":"2024-04-20T02:50:27.638986+0000","last_fullsized":"2024-04-20T02:50:27.638986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:29.030331+0000","last_change":"2024-04-20T02:50:28.091938+0000","last_active":"2024-04-20T02:50:29.030331+0000","last_peered":"2024-04-20T02:50:29.030331+0000","last_clean":"2024-04-20T02:50:29.030331+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:50:29.030331+0000","last_undegraded":"2024-04-20T02:50:29.030331+0000","last_fullsized":"2024-04-20T02:50:29.030331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:34.990718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:29.030307+0000","last_change":"2024-04-20T02:50:28.092115+0000","last_active":"2024-04-20T02:50:29.030307+0000","last_peered":"2024-04-20T02:50:29.030307+0000","last_clean":"2024-04-20T02:50:29.030307+0000","last_became_active":"2024-04-20T02:50:28.091570+0000","last_became_peered":"2024-04-20T02:50:28.091570+0000","last_unstale":"2024-04-20T02:50:29.030307+0000","last_undegraded":"2024-04-20T02:50:29.030307+0000","last_fullsized":"2024-04-20T02:50:29.030307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:27.403210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:27.636042+0000","last_change":"2024-04-20T02:50:19.923431+0000","last_active":"2024-04-20T02:50:27.636042+0000","last_peered":"2024-04-20T02:50:27.636042+0000","last_clean":"2024-04-20T02:50:27.636042+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:50:27.636042+0000","last_undegraded":"2024-04-20T02:50:27.636042+0000","last_fullsized":"2024-04-20T02:50:27.636042+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:50:17.569030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.655607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.639459+0000","last_change":"2024-04-20T02:50:27.638361+0000","last_active":"2024-04-20T02:50:28.639459+0000","last_peered":"2024-04-20T02:50:28.639459+0000","last_clean":"2024-04-20T02:50:28.639459+0000","last_became_active":"2024-04-20T02:50:27.637907+0000","last_became_peered":"2024-04-20T02:50:27.637907+0000","last_unstale":"2024-04-20T02:50:28.639459+0000","last_undegraded":"2024-04-20T02:50:28.639459+0000","last_fullsized":"2024-04-20T02:50:28.639459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:41.417237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.091742+0000","last_change":"2024-04-20T02:50:28.091742+0000","last_active":"2024-04-20T02:50:28.091742+0000","last_peered":"2024-04-20T02:50:28.091742+0000","last_clean":"2024-04-20T02:50:28.091742+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:50:28.091742+0000","last_undegraded":"2024-04-20T02:50:28.091742+0000","last_fullsized":"2024-04-20T02:50:28.091742+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:32.326186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.639461+0000","last_change":"2024-04-20T02:50:27.638420+0000","last_active":"2024-04-20T02:50:28.639461+0000","last_peered":"2024-04-20T02:50:28.639461+0000","last_clean":"2024-04-20T02:50:28.639461+0000","last_became_active":"2024-04-20T02:50:27.638143+0000","last_became_peered":"2024-04-20T02:50:27.638143+0000","last_unstale":"2024-04-20T02:50:28.639461+0000","last_undegraded":"2024-04-20T02:50:28.639461+0000","last_fullsized":"2024-04-20T02:50:28.639461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.959545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-20T02:50:28.091519+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:50:28.091519+0000","last_peered":"2024-04-20T02:50:28.091519+0000","last_clean":"2024-04-20T02:50:28.091519+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:50:28.091519+0000","last_undegraded":"2024-04-20T02:50:28.091519+0000","last_fullsized":"2024-04-20T02:50:28.091519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509448,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":585821,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]}]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":56893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":52265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":581193,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509444,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":585821,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":172032,"data_stored":56893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:50:31.913 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-20T02:50:31.913 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-20T02:50:32.240 INFO:teuthology.orchestra.run.smithi105.stdout:epoch 20 2024-04-20T02:50:32.240 INFO:teuthology.orchestra.run.smithi105.stdout:fsid a524a365-eaec-4ebd-aa4d-0d76d55a252f 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:created 2024-04-20T02:50:07.356902+0000 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:modified 2024-04-20T02:50:28.629861+0000 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:crush_version 4 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:full_ratio 0.95 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:backfillfull_ratio 0.9 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:nearfull_ratio 0.85 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:require_min_compat_client luminous 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:min_compat_client jewel 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:require_osd_release reef 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:stretch_mode_enabled false 2024-04-20T02:50:32.241 INFO:teuthology.orchestra.run.smithi105.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-04-20T02:50:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 application rbd read_balance_score 2.00 2024-04-20T02:50:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:max_osd 8 2024-04-20T02:50:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.105:6812/43330 v2:172.21.15.105:6813/43330 exists,up 8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6 2024-04-20T02:50:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.105:6804/43331 v2:172.21.15.105:6805/43331 exists,up c5ae8a64-fc7c-47ab-af6a-4b762a840645 2024-04-20T02:50:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.105:6800/43326 v2:172.21.15.105:6801/43326 exists,up 336e852e-5fa6-4ac0-b053-d31ad40d7c4e 2024-04-20T02:50:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.105:6808/43329 v2:172.21.15.105:6809/43329 exists,up 695cab3a-74d8-4046-ac61-2656dc0d893e 2024-04-20T02:50:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.138:6808/43330 v2:172.21.15.138:6809/43330 exists,up 1d41da44-d894-40ac-acaa-aa0df15296c6 2024-04-20T02:50:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.138:6802/43328 v2:172.21.15.138:6804/43328 exists,up 3e1c33fa-6c8c-4898-99bc-462ecef25f30 2024-04-20T02:50:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.138:6800/43329 v2:172.21.15.138:6801/43329 exists,up fd1fc124-7cf8-4628-9076-1f27d6385676 2024-04-20T02:50:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.138:6812/43351 v2:172.21.15.138:6813/43351 exists,up 53ae8193-5c3e-4698-aacd-df2ad1db9143 2024-04-20T02:50:32.255 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.105:6812/43330 v2:172.21.15.105:6813/43330 exists,up 8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6', 'osd.1 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.105:6804/43331 v2:172.21.15.105:6805/43331 exists,up c5ae8a64-fc7c-47ab-af6a-4b762a840645', 'osd.2 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.105:6800/43326 v2:172.21.15.105:6801/43326 exists,up 336e852e-5fa6-4ac0-b053-d31ad40d7c4e', 'osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.105:6808/43329 v2:172.21.15.105:6809/43329 exists,up 695cab3a-74d8-4046-ac61-2656dc0d893e', 'osd.4 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.138:6808/43330 v2:172.21.15.138:6809/43330 exists,up 1d41da44-d894-40ac-acaa-aa0df15296c6', 'osd.5 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.138:6802/43328 v2:172.21.15.138:6804/43328 exists,up 3e1c33fa-6c8c-4898-99bc-462ecef25f30', 'osd.6 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.138:6800/43329 v2:172.21.15.138:6801/43329 exists,up fd1fc124-7cf8-4628-9076-1f27d6385676', 'osd.7 up in weight 1 up_from 15 up_thru 18 down_at 0 last_clean_interval [0,0) v2:172.21.15.138:6812/43351 v2:172.21.15.138:6813/43351 exists,up 53ae8193-5c3e-4698-aacd-df2ad1db9143'] 2024-04-20T02:50:32.255 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-20T02:50:32.378 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:32.378 INFO:teuthology.orchestra.run.smithi105.stdout: "name": "mon.a", 2024-04-20T02:50:32.378 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster": "ceph", 2024-04-20T02:50:32.378 INFO:teuthology.orchestra.run.smithi105.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-20T02:50:32.378 INFO:teuthology.orchestra.run.smithi105.stdout: "admin_socket_mode": "", 2024-04-20T02:50:32.378 INFO:teuthology.orchestra.run.smithi105.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "auth_client_required": "cephx, none", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "auth_cluster_required": "cephx", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "auth_debug": "true", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "auth_service_required": "cephx", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "auth_supported": "cephx", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_aio": "true", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_aio_poll_ms": "250", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_aio_reap_max": "16", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_async_discard": "false", 2024-04-20T02:50:32.379 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_block_size": "4096", 2024-04-20T02:50:32.380 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_debug_aio": "false", 2024-04-20T02:50:32.380 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-20T02:50:32.380 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-20T02:50:32.380 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_debug_inflight_ios": "false", 2024-04-20T02:50:32.380 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_enable_discard": "false", 2024-04-20T02:50:32.380 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_flock_retry": "3", 2024-04-20T02:50:32.380 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-20T02:50:32.380 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_inject_crash": "0", 2024-04-20T02:50:32.380 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-20T02:50:32.380 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_ioring": "false", 2024-04-20T02:50:32.381 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_ioring_hipri": "false", 2024-04-20T02:50:32.381 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-20T02:50:32.381 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-20T02:50:32.381 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-20T02:50:32.381 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-20T02:50:32.381 INFO:teuthology.orchestra.run.smithi105.stdout: "bdev_type": "", 2024-04-20T02:50:32.381 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_alloc_size": "1048576", 2024-04-20T02:50:32.381 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_allocator": "hybrid", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_buffered_io": "true", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_check_for_zeros": "false", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_compact_log_sync": "false", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_max_log_runway": "4194304", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_max_prefetch": "1048576", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_min_flush_size": "524288", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_min_log_runway": "1048576", 2024-04-20T02:50:32.382 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_replay_recovery": "false", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_sync_write": "false", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_allocation_from_file": "true", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_allocator": "hybrid", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-20T02:50:32.383 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-20T02:50:32.384 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-20T02:50:32.384 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-20T02:50:32.384 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_block_create": "true", 2024-04-20T02:50:32.384 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_block_db_create": "false", 2024-04-20T02:50:32.384 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_block_db_path": "", 2024-04-20T02:50:32.384 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_block_db_size": "0", 2024-04-20T02:50:32.384 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_block_path": "", 2024-04-20T02:50:32.384 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_block_preallocate_file": "false", 2024-04-20T02:50:32.384 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_block_size": "107374182400", 2024-04-20T02:50:32.384 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_block_wal_create": "false", 2024-04-20T02:50:32.385 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_block_wal_path": "", 2024-04-20T02:50:32.385 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_block_wal_size": "100663296", 2024-04-20T02:50:32.385 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_bluefs": "true", 2024-04-20T02:50:32.385 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-20T02:50:32.385 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-20T02:50:32.385 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-20T02:50:32.385 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-20T02:50:32.385 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_autotune": "true", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_size": "0", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-20T02:50:32.386 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_type": "2q", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_clone_cow": "true", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_compression_mode": "none", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_csum_type": "crc32c", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-20T02:50:32.387 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_freelist": "false", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_inject_bug21040": "false", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-20T02:50:32.388 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_small_allocations": "0", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_default_buffered_read": "true", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_default_buffered_write": "false", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fail_eio": "false", 2024-04-20T02:50:32.389 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_on_mount": "false", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_on_umount": "false", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-20T02:50:32.390 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-20T02:50:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-20T02:50:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-20T02:50:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_ignore_data_csum": "false", 2024-04-20T02:50:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-20T02:50:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-20T02:50:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-20T02:50:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-20T02:50:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_log_op_age": "5.000000", 2024-04-20T02:50:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_max_alloc_size": "0", 2024-04-20T02:50:32.391 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_max_blob_size": "0", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_max_deferred_txc": "32", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_min_alloc_size": "0", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_nid_prealloc": "1024", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_retry_disk_reads": "3", 2024-04-20T02:50:32.392 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_rocksdb_cf": "true", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_spdk_mem": "512", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-20T02:50:32.393 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_tracing": "false", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_zero_block_detection": "false", 2024-04-20T02:50:32.394 INFO:teuthology.orchestra.run.smithi105.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-20T02:50:32.395 INFO:teuthology.orchestra.run.smithi105.stdout: "cephx_cluster_require_signatures": "false", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "cephx_cluster_require_version": "2", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "cephx_require_signatures": "false", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "cephx_require_version": "2", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "cephx_service_require_signatures": "false", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "cephx_service_require_version": "2", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "cephx_sign_messages": "true", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "chdir": "", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_acl_type": "", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_asio_thread_count": "2", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_cache_mid": "0.750000", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_cache_size": "16384", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_caps_release_delay": "5", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_check_pool_perm": "true", 2024-04-20T02:50:32.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_debug_force_sync_read": "false", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_debug_getattr_caps": "false", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_debug_inject_tick_delay": "0", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_die_on_failed_remount": "false", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_dirsize_rbytes": "true", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_force_lazyio": "false", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_fs": "", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_inject_release_failure": "false", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_max_inline_size": "4096", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_mds_namespace": "", 2024-04-20T02:50:32.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_metadata": "", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_mount_gid": "-1", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_mount_timeout": "300", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_mount_uid": "-1", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_mountpoint": "/", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_notify_timeout": "10", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_oc": "true", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_oc_max_dirty": "104857600", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_oc_max_objects": "1000", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_oc_size": "209715200", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_oc_target_dirty": "8388608", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_permissions": "true", 2024-04-20T02:50:32.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_quota": "true", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_quota_df": "true", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_readahead_max_bytes": "0", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_readahead_max_periods": "4", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_readahead_min": "131072", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_reconnect_stale": "false", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_shutdown_timeout": "30", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_snapdir": ".snap", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_tick_interval": "1", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_trace": "", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_try_dentry_invalidate": "false", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_use_faked_inos": "false", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "client_use_random_mds": "false", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "clog_to_graylog": "false", 2024-04-20T02:50:32.399 INFO:teuthology.orchestra.run.smithi105.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "clog_to_graylog_port": "12201", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "clog_to_monitors": "default=true", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "clog_to_syslog": "false", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "clog_to_syslog_level": "info", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "-", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_network": "", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_network_interface": "", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "compressor_zlib_isal": "false", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "compressor_zlib_level": "5", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "compressor_zlib_winsize": "-15", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "compressor_zstd_level": "1", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-20T02:50:32.400 INFO:teuthology.orchestra.run.smithi105.stdout: "crimson_alien_op_num_threads": "6", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "crimson_osd_obc_lru_size": "10", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "crimson_seastar_smp": "1", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_location": "", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_location_hook": "", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_location_hook_timeout": "10", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "daemonize": "false", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "daos_pool": "tank", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_alienstore": "0/5", 2024-04-20T02:50:32.401 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_allow_any_pool_priority": "false", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_asok": "1/5", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_asok_assert_abort": "false", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_asserts_on_shutdown": "true", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_auth": "20/20", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_bdev": "1/3", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_bluefs": "1/5", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_bluestore": "1/5", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_buffer": "0/1", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_ceph_exporter": "1/5", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_cephfs_mirror": "0/5", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_cephsqlite": "0/5", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_client": "0/5", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_compressor": "1/5", 2024-04-20T02:50:32.402 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_context": "0/1", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_crush": "1/1", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_crypto": "1/5", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_cyanstore": "0/5", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_deliberately_leak_memory": "false", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_disable_randomized_ping": "false", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_dpdk": "1/5", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_eventtrace": "1/5", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_filer": "0/1", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_filestore": "1/3", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_finisher": "1/1", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_fuse": "1/5", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_heartbeat_testing_span": "0", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_heartbeatmap": "1/5", 2024-04-20T02:50:32.403 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_javaclient": "1/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_journal": "1/3", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_journaler": "0/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_kstore": "1/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_leveldb": "4/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_lockdep": "0/1", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_mclock": "1/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_mds": "1/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_mds_balancer": "1/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_mds_locker": "1/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_mds_log": "1/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_mds_log_expire": "1/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_mds_migrator": "1/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_memstore": "1/5", 2024-04-20T02:50:32.404 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_mgr": "2/5", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_mgrc": "1/5", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_mon": "20/20", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_monc": "0/10", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_ms": "1/1", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_none": "0/5", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_objclass": "0/5", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_objectcacher": "0/5", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_objecter": "0/1", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_optracker": "0/5", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_osd": "1/5", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_paxos": "20/20", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_perfcounter": "1/5", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_prioritycache": "1/5", 2024-04-20T02:50:32.405 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rados": "0/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rbd": "0/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rbd_mirror": "0/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rbd_pwl": "0/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rbd_replay": "0/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_refs": "0/0", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_reserver": "1/1", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rgw": "1/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rgw_access": "1/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rgw_datacache": "1/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rgw_dbstore": "1/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rgw_flight": "1/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rgw_sync": "1/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_rocksdb": "4/5", 2024-04-20T02:50:32.406 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_backref": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_cache": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_cleaner": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_device": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_epm": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_journal": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_lba": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_odata": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_omap": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_onode": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_t": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_seastore_tm": "0/5", 2024-04-20T02:50:32.407 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_striper": "0/1", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_test": "0/5", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_throttle": "1/1", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_timer": "0/1", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "debug_tp": "0/5", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "device_failure_prediction_mode": "none", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "err_to_graylog": "false", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "err_to_journald": "false", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "err_to_stderr": "true", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "err_to_syslog": "false", 2024-04-20T02:50:32.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event_tracing": "false", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "exporter_addr": "0.0.0.0", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "exporter_http_port": "9926", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "exporter_prio_limit": "5", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "exporter_sort_metrics": "true", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "exporter_stats_period": "5", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "fake_statfs_for_testing": "0", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "fatal_signal_handlers": "true", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "filer_max_purge_ops": "10", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "filer_max_truncate_ops": "128", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_apply_finisher_threads": "1", 2024-04-20T02:50:32.409 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_blackhole": "false", 2024-04-20T02:50:32.410 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_btrfs_clone_range": "true", 2024-04-20T02:50:32.410 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_btrfs_snap": "true", 2024-04-20T02:50:32.410 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_caller_concurrency": "10", 2024-04-20T02:50:32.410 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_collect_device_partition_information": "true", 2024-04-20T02:50:32.410 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_commit_timeout": "600.000000", 2024-04-20T02:50:32.410 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_debug_inject_read_err": "false", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_debug_omap_check": "false", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_debug_verify_split": "false", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_dump_file": "", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_fadvise": "true", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_fail_eio": "true", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_fd_cache_shards": "16", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_fd_cache_size": "128", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_fiemap": "false", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_fiemap_threshold": "4096", 2024-04-20T02:50:32.411 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_inject_stall": "0", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_journal_parallel": "false", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_journal_trailing": "false", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_journal_writeahead": "false", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_kill_at": "0", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_inline_xattrs": "0", 2024-04-20T02:50:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_xattr_value_size": "0", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_merge_threshold": "-10", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_odsync_write": "false", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_omap_backend": "rocksdb", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_omap_backend_path": "", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-20T02:50:32.413 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-20T02:50:32.414 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-20T02:50:32.414 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_op_thread_timeout": "60", 2024-04-20T02:50:32.414 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_op_threads": "2", 2024-04-20T02:50:32.414 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_punch_hole": "false", 2024-04-20T02:50:32.414 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-20T02:50:32.414 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-20T02:50:32.414 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-20T02:50:32.414 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-20T02:50:32.414 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-20T02:50:32.414 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_queue_max_ops": "50", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_seek_data_hole": "false", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_sloppy_crc": "false", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_splice": "false", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_split_multiple": "2", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_split_rand_factor": "20", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_update_to": "1000", 2024-04-20T02:50:32.415 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_enable": "true", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_xfs_extsize": "false", 2024-04-20T02:50:32.416 INFO:teuthology.orchestra.run.smithi105.stdout: "filestore_zfs_snap": "false", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fio_dir": "/tmp/fio", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fsid": "a524a365-eaec-4ebd-aa4d-0d76d55a252f", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_allow_other": "true", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_atomic_o_trunc": "true", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_big_writes": "true", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_debug": "false", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_default_permissions": "false", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_disable_pagecache": "false", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_max_write": "0", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_multithreaded": "true", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_require_active_mds": "true", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_set_user_groups": "true", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_splice_move": "true", 2024-04-20T02:50:32.417 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_splice_read": "true", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_splice_write": "true", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "fuse_use_invalidate_cb": "true", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "gss_target_name": "ceph", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_file": "", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_inject_failure": "0", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_interval": "5", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "host": "smithi105", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-20T02:50:32.418 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "inject_early_sigterm": "false", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "jaeger_agent_port": "6799", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "jaeger_tracing_enable": "false", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_aio": "true", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_align_min_size": "65536", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_block_align": "true", 2024-04-20T02:50:32.419 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_block_size": "4096", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_dio": "true", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_discard": "false", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_force_aio": "false", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_ignore_corruption": "false", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_max_write_bytes": "10485760", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_max_write_entries": "100", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_replay_from": "0", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_write_header_frequency": "0", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journal_zero_on_create": "false", 2024-04-20T02:50:32.420 INFO:teuthology.orchestra.run.smithi105.stdout: "journaler_prefetch_periods": "10", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "journaler_prezero_periods": "5", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "journaler_write_head_interval": "15", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "key": "", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "keyfile": "", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_backend": "rocksdb", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_default_stripe_size": "65536", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_fsck_on_mount": "false", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_max_bytes": "67108864", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_max_ops": "512", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_nid_prealloc": "1024", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_onode_map_size": "1024", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-20T02:50:32.421 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_sync_submit_transaction": "false", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "kstore_sync_transaction": "false", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "leveldb_block_size": "65536", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "leveldb_bloom_size": "0", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "leveldb_cache_size": "536870912", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "leveldb_compact_on_mount": "false", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "leveldb_compression": "false", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "leveldb_log": "", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "leveldb_log_to_ceph_log": "true", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "leveldb_max_open_files": "0", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "leveldb_paranoid": "false", 2024-04-20T02:50:32.422 INFO:teuthology.orchestra.run.smithi105.stdout: "leveldb_write_buffer_size": "33554432", 2024-04-20T02:50:32.423 INFO:teuthology.orchestra.run.smithi105.stdout: "librados_thread_count": "2", 2024-04-20T02:50:32.423 INFO:teuthology.orchestra.run.smithi105.stdout: "lockdep": "false", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "lockdep_force_backtrace": "false", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_coarse_timestamps": "true", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_flush_on_exit": "false", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_graylog_host": "127.0.0.1", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_graylog_port": "12201", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_max_new": "1000", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_max_recent": "10000", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_stderr_prefix": "", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_stop_at_utilization": "0.970000", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_file": "true", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_graylog": "false", 2024-04-20T02:50:32.424 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_journald": "false", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_stderr": "false", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "log_to_syslog": "false", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "max_rotating_auth_attempts": "10", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_action_on_write_error": "1", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_alternate_name_max": "8192", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_asio_thread_count": "2", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_export_pin": "true", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_fragment_dirs": "true", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_fragment_interval": "5", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-20T02:50:32.425 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-20T02:50:32.426 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_interval": "10", 2024-04-20T02:50:32.426 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_max": "-1", 2024-04-20T02:50:32.426 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_max_until": "-1", 2024-04-20T02:50:32.426 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_merge_size": "50", 2024-04-20T02:50:32.426 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_midchunk": "0.300000", 2024-04-20T02:50:32.426 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-20T02:50:32.426 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_min_start": "0.200000", 2024-04-20T02:50:32.426 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_minchunk": "0.001000", 2024-04-20T02:50:32.426 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_mode": "0", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_need_max": "1.200000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_need_min": "0.800000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_split_bits": "3", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_split_size": "10000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_target_decay": "10.000000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_beacon_grace": "15.000000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_beacon_interval": "4.000000", 2024-04-20T02:50:32.427 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache_mid": "0.700000", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache_release_free_interval": "10", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache_reservation": "0.050000", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache_trim_interval": "1", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache_trim_threshold": "262144", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_client_prealloc_inos": "1000", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_connect_bootstrapping": "false", 2024-04-20T02:50:32.428 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_damage_table_max_entries": "10000", 2024-04-20T02:50:32.429 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-20T02:50:32.429 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_debug_auth_pins": "false", 2024-04-20T02:50:32.429 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_debug_frag": "false", 2024-04-20T02:50:32.429 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_debug_scatterstat": "false", 2024-04-20T02:50:32.429 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_debug_subtrees": "false", 2024-04-20T02:50:32.429 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_decay_halflife": "5.000000", 2024-04-20T02:50:32.429 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_default_dir_hash": "2", 2024-04-20T02:50:32.429 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_defer_session_stale": "true", 2024-04-20T02:50:32.429 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_deny_all_reconnect": "false", 2024-04-20T02:50:32.429 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_dir_keys_per_op": "16384", 2024-04-20T02:50:32.430 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_dir_max_commit_size": "10", 2024-04-20T02:50:32.430 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_dir_max_entries": "0", 2024-04-20T02:50:32.430 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_dir_prefetch": "true", 2024-04-20T02:50:32.430 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-20T02:50:32.430 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-20T02:50:32.430 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_dump_cache_on_map": "false", 2024-04-20T02:50:32.430 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-20T02:50:32.430 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-20T02:50:32.430 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_early_reply": "true", 2024-04-20T02:50:32.430 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_enable_op_tracker": "true", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_enforce_unique_name": "true", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_export_ephemeral_random": "true", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-20T02:50:32.431 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-20T02:50:32.432 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_health_summarize_threshold": "10", 2024-04-20T02:50:32.432 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-20T02:50:32.432 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-20T02:50:32.432 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_inject_health_dummy": "false", 2024-04-20T02:50:32.432 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_inject_migrator_session_race": "false", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_join_fs": "", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_journal_format": "1", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_create_at": "0", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_export_at": "0", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_import_at": "0", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_journal_at": "0", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_journal_expire_at": "0", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_journal_replay_at": "0", 2024-04-20T02:50:32.433 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_link_at": "0", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_mdstable_at": "0", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_openc_at": "0", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_rename_at": "0", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log_events_per_segment": "1024", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log_max_events": "-1", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log_max_segments": "128", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log_pause": "false", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log_segment_size": "0", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log_warn_factor": "2.000000", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_max_caps_per_client": "1048576", 2024-04-20T02:50:32.434 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_max_completed_flushes": "100000", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_max_completed_requests": "100000", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_max_export_size": "20971520", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_max_file_recover": "32", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_max_purge_files": "64", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_max_purge_ops": "8192", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_max_snaps_per_dir": "100", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_metrics_update_interval": "2", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_min_caps_per_client": "100", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_min_caps_working_set": "10000", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_numa_node": "-1", 2024-04-20T02:50:32.435 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_op_complaint_time": "30.000000", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_op_history_duration": "600", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_op_history_size": "20", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_op_history_slow_op_size": "20", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_op_log_threshold": "5", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_ping_grace": "15", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_ping_interval": "5", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_recall_max_caps": "30000", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-20T02:50:32.436 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_recall_warning_threshold": "262144", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_replay_interval": "1.000000", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_root_ino_gid": "0", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_root_ino_uid": "0", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-20T02:50:32.437 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_shutdown_check": "0", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_skip_ino": "0", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_snap_max_uid": "4294967294", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_snap_min_uid": "0", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_snap_rstat": "false", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_standby_replay_damaged": "false", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_symlink_recovery": "true", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_thrash_exports": "0", 2024-04-20T02:50:32.438 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_thrash_fragments": "0", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_tick_interval": "5.000000", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_valgrind_exit": "false", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_verify_backtrace": "1", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_verify_scatter": "false", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_wipe_sessions": "false", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool_debug": "false", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "memstore_device_bytes": "1073741824", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "memstore_page_set": "false", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "memstore_page_size": "65536", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_client_bytes": "134217728", 2024-04-20T02:50:32.439 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_client_messages": "512", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_disabled_modules": "", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_max_pg_num_change": "128", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_mds_bytes": "134217728", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_mds_messages": "128", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_mon_bytes": "134217728", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_mon_messages": "128", 2024-04-20T02:50:32.440 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_osd_bytes": "536870912", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_osd_messages": "8192", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_pool": "true", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_standby_modules": "true", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_stats_period": "5", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_stats_threshold": "5", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_tick_period": "2", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_allow_pool_delete": "true", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_allow_pool_size_one": "true", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_auth_validate_all_caps": "true", 2024-04-20T02:50:32.441 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_bytes": "104857600", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_directed_command_retry": "5", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_hunt_parallel": "3", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_log_interval": "1.000000", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_ping_interval": "10.000000", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-20T02:50:32.442 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_file_level": "debug", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_to_file": "true", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_to_journald": "false", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_compact_on_bootstrap": "false", 2024-04-20T02:50:32.443 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_compact_on_start": "false", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_compact_on_trim": "true", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_cpu_threads": "4", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_crush_min_required_version": "hammer", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_daemon_bytes": "419430400", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_data_avail_crit": "5", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_data_avail_warn": "5", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_data_size_warn": "16106127360", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-20T02:50:32.444 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_dump_json": "false", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_dump_transactions": "false", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_extra_checks": "false", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_no_require_quincy": "false", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_no_require_reef": "false", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_down_mkfs_grace": "120", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_election_default_strategy": "3", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_election_timeout": "5.000000", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-20T02:50:32.445 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_elector_ping_divisor": "2", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_enable_op_tracker": "true", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_fake_pool_delete": "false", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_force_quorum_join": "false", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_globalid_prealloc": "10000", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_health_detail_to_clog": "false", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_health_log_update_period": "5", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_health_max_detail": "50", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_health_to_clog": "true", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_health_to_clog_interval": "600", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_host": "172.21.15.105,[v2:172.21.15.105:3301,v1:172.21.15.105:6790],172.21.15.138", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_host_override": "", 2024-04-20T02:50:32.446 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_initial_members": "", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_lease": "5.000000", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_log_full_interval": "50", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_log_max": "10000", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_log_max_summary": "50", 2024-04-20T02:50:32.447 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_max_log_epochs": "500", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_max_osd": "10000", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_max_pg_per_osd": "10000", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_max_pool_pg_num": "65536", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_mds_force_trim_to": "0", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_mds_skip_sanity": "false", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_memory_autotune": "true", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_memory_target": "2147483648", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_mgr_beacon_grace": "30", 2024-04-20T02:50:32.448 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_mgr_digest_period": "5", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_mgr_inactive_grace": "60", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_min_osdmap_epochs": "50", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_op_complaint_time": "30", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_op_history_duration": "600", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_op_history_size": "20", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_op_history_slow_op_size": "20", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_op_log_threshold": "5", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-20T02:50:32.449 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_auto_mark_in": "false", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_cache_size": "500", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_down_out_interval": "600", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_force_trim_to": "0", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-20T02:50:32.450 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_min_down_reporters": "2", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_report_timeout": "900", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-20T02:50:32.451 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_pg_stuck_threshold": "60", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_probe_timeout": "2.000000", 2024-04-20T02:50:32.452 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_reweight_max_change": "0.050000", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_reweight_max_osds": "4", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_scrub_interval": "300", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_scrub_max_keys": "100", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_scrub_timeout": "300", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_session_timeout": "300", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_smart_report_timeout": "5", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_stat_smooth_intervals": "6", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-20T02:50:32.453 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_stretch_pool_min_size": "2", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_stretch_pool_size": "4", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_sync_debug": "false", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_sync_provider_kill_at": "0", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_sync_requester_kill_at": "0", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_sync_timeout": "60.000000", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_target_pg_per_osd": "100", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_tick_interval": "5", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_timecheck_interval": "300.000000", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-20T02:50:32.454 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_use_min_delay_socket": "false", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_older_version_delay": "604800", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_misplaced": "false", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_older_version": "true", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-20T02:50:32.455 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "monmap": "", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-20T02:50:32.456 INFO:teuthology.orchestra.run.smithi105.stdout: "motr_tracing_enabled": "false", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_op_threads": "3", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_cm": "false", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_device_name": "", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_dscp": "96", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_local_gid": "", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_port_num": "1", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-20T02:50:32.457 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_sl": "3", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_support_srq": "true", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_rdma_type": "ib", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_async_reap_threshold": "5", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_bind_before_connect": "false", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_bind_ipv4": "true", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_bind_ipv6": "false", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_bind_msgr1": "false", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_bind_msgr2": "true", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_bind_port_max": "7568", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_bind_port_min": "6800", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_bind_retry_count": "3", 2024-04-20T02:50:32.458 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_bind_retry_delay": "5", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_blackhole_client": "false", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_blackhole_mds": "false", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_blackhole_mgr": "false", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_blackhole_mon": "false", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_blackhole_osd": "false", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_client_mode": "crc secure", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_cluster_mode": "crc secure", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_cluster_type": "", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_compress_secure": "false", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_connection_idle_timeout": "900", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_connection_ready_timeout": "10", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_crc_data": "true", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_crc_header": "true", 2024-04-20T02:50:32.459 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_die_on_bad_msg": "false", 2024-04-20T02:50:32.460 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_die_on_bug": "true", 2024-04-20T02:50:32.460 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_die_on_old_message": "true", 2024-04-20T02:50:32.460 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_die_on_skipped_message": "false", 2024-04-20T02:50:32.460 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-20T02:50:32.460 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-20T02:50:32.460 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_coremask": "0xF", 2024-04-20T02:50:32.460 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-20T02:50:32.460 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-20T02:50:32.460 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_enable_tso": "true", 2024-04-20T02:50:32.460 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_hugepages": "", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_lro": "true", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_memory_channel": "4", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_pmd": "", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_port_id": "0", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-20T02:50:32.461 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_dump_on_send": "false", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_initial_backoff": "0.200000", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_inject_delay_max": "1.000000", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_inject_delay_type": "", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_inject_network_congestion": "0", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_inject_socket_failures": "2500", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_learn_addr_from_peer": "true", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_max_accept_failures": "4", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_max_backoff": "15.000000", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_mon_client_mode": "secure crc", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_mon_service_mode": "secure crc", 2024-04-20T02:50:32.462 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_osd_compress_min_size": "1024", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_osd_compress_mode": "none", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_pq_min_cost": "65536", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_public_type": "", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_service_mode": "crc secure", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_tcp_listen_backlog": "512", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_tcp_nodelay": "true", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_tcp_rcvbuf": "0", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "ms_type": "async", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "no_config_file": "false", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter_completion_locks_per_session": "32", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-20T02:50:32.463 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter_inflight_ops": "1024", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter_tick_interval": "5.000000", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter_timeout": "10.000000", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "objectstore_blackhole": "false", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "openssl_engine_opts": "", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_agent_delay_time": "5.000000", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_agent_hist_halflife": "1000", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_agent_max_low_ops": "2", 2024-04-20T02:50:32.464 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_agent_max_ops": "4", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_agent_slop": "0.020000", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_asio_thread_count": "2", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_async_recovery_min_cost": "1", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_backfill_scan_max": "512", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_backfill_scan_min": "64", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_backoff_on_degraded": "false", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_backoff_on_peering": "false", 2024-04-20T02:50:32.465 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_backoff_on_unfound": "true", 2024-04-20T02:50:32.466 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_beacon_report_interval": "300", 2024-04-20T02:50:32.466 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bench_duration": "30", 2024-04-20T02:50:32.466 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-20T02:50:32.466 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bench_max_block_size": "67108864", 2024-04-20T02:50:32.466 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-20T02:50:32.466 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_blkin_trace_all": "false", 2024-04-20T02:50:32.466 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_check_for_log_corruption": "false", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_class_update_on_start": "true", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_client_message_cap": "256", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_client_message_size_cap": "524288000", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_client_op_priority": "63", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_client_watch_timeout": "30", 2024-04-20T02:50:32.467 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_command_max_records": "256", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_compact_on_start": "false", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_crush_update_on_start": "true", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_crush_update_weight_set": "true", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_feed_pullee": "-1", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-20T02:50:32.468 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_misdirected_ops": "false", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_no_acting_change": "false", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_no_purge_strays": "false", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_op_order": "false", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_shutdown": "false", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-20T02:50:32.469 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_verify_snaps": "false", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_deep_scrub_keys": "1024", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_deep_scrub_stride": "524288", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-20T02:50:32.470 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_default_notify_timeout": "30", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_delete_sleep": "0.000000", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_discard_disconnected_ops": "true", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_enable_op_tracker": "true", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_fast_info": "true", 2024-04-20T02:50:32.471 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_fast_shutdown": "true", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_function_tracing": "false", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_heartbeat_grace": "20", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_heartbeat_interval": "6", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_heartbeat_min_peers": "10", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_heartbeat_min_size": "2000", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_heartbeat_stale": "600", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-20T02:50:32.472 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_hit_set_max_size": "100000", 2024-04-20T02:50:32.473 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_hit_set_min_size": "1000", 2024-04-20T02:50:32.473 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-20T02:50:32.473 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-20T02:50:32.473 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-20T02:50:32.473 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-20T02:50:32.473 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-20T02:50:32.473 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-20T02:50:32.473 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_journal_size": "5120", 2024-04-20T02:50:32.473 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_kill_backfill_at": "0", 2024-04-20T02:50:32.473 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_map_cache_size": "50", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_map_dedup": "true", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_map_message_max": "40", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_map_share_max_epochs": "40", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_attr_name_len": "100", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_attr_size": "0", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_backfills": "1", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_markdown_count": "5", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_markdown_period": "600", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_object_name_len": "2048", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_object_namespace_len": "256", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_object_size": "134217728", 2024-04-20T02:50:32.474 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_pg_blocked_by": "16", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_pg_log_entries": "2", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_pgls": "1024", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_push_cost": "8388608", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_push_objects": "10", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_scrubs": "3", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_trimming_pgs": "2", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_write_op_reply_len": "64", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_max_write_size": "90", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-20T02:50:32.475 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_profile": "balanced", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-20T02:50:32.476 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_memory_base": "805306368", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_memory_cache_min": "134217728", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_memory_target": "4294967296", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_memory_target_autotune": "false", 2024-04-20T02:50:32.477 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_min_pg_log_entries": "1", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_min_recovery_priority": "0", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mon_report_interval": "5", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_num_cache_shards": "32", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_num_op_tracker_shard": "32", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_numa_auto_affinity": "true", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_numa_node": "-1", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_numa_prefer_iface": "true", 2024-04-20T02:50:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_objecter_finishers": "1", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_objectstore": "bluestore", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_objectstore_fuse": "false", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_objectstore_tracing": "false", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_complaint_time": "30.000000", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_history_duration": "600", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_history_size": "20", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_history_slow_op_size": "20", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_log_threshold": "5", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_num_shards": "0", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_num_shards_hdd": "5", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_num_shards_ssd": "8", 2024-04-20T02:50:32.479 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_pq_min_cost": "65536", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_queue_cut_off": "high", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_op_thread_timeout": "15", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_open_classes_on_start": "true", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_os_flags": "0", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_peering_op_priority": "255", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pg_delete_cost": "1048576", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pg_delete_priority": "5", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-20T02:50:32.480 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pg_log_trim_max": "10000", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pg_log_trim_min": "0", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pg_stat_report_interval_max": "500", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_crimson": "false", 2024-04-20T02:50:32.481 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_flags": "0", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_min_size": "2", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-20T02:50:32.482 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_pg_num": "32", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_pgp_num": "0", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_size": "3", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_default_type": "replicated", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_push_per_object_cost": "1000", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recover_clone_overlap": "true", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_cost": "20971520", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_max_active": "0", 2024-04-20T02:50:32.483 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_max_single_start": "1", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_op_priority": "3", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_priority": "5", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_sleep": "0.000000", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_repair_during_recovery": "false", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_requested_scrub_priority": "120", 2024-04-20T02:50:32.484 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_auto_repair": "false", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_begin_hour": "0", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_begin_week_day": "0", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_chunk_max": "25", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_chunk_min": "5", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_cost": "52428800", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_during_recovery": "false", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_end_hour": "0", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_end_week_day": "0", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_event_cost": "4096", 2024-04-20T02:50:32.485 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-20T02:50:32.486 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-20T02:50:32.486 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_invalid_stats": "true", 2024-04-20T02:50:32.486 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-20T02:50:32.486 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-20T02:50:32.486 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_max_preemptions": "5", 2024-04-20T02:50:32.487 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-20T02:50:32.487 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_priority": "5", 2024-04-20T02:50:32.487 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_reservation_timeout": "5000", 2024-04-20T02:50:32.487 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_sleep": "0.000000", 2024-04-20T02:50:32.487 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_scrub_slow_reservation_response": "2200", 2024-04-20T02:50:32.487 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-20T02:50:32.487 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-20T02:50:32.487 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-20T02:50:32.487 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_skip_data_digest": "false", 2024-04-20T02:50:32.487 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_smart_report_timeout": "5", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_snap_trim_cost": "1048576", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_snap_trim_priority": "5", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_target_transaction_size": "30", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-20T02:50:32.488 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_tracing": "false", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_use_stale_snap": "false", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "osdc_blkin_trace_all": "false", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_kill_at": "0", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_max_join_drift": "10", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_min": "500", 2024-04-20T02:50:32.489 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_min_wait": "0.050000", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_propose_interval": "1.000000", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_service_trim_max": "500", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_service_trim_min": "10", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_stash_full_interval": "25", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_trim_max": "500", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "paxos_trim_min": "250", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "perf": "true", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "-", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrv": "", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "public_bind_addr": "-", 2024-04-20T02:50:32.490 INFO:teuthology.orchestra.run.smithi105.stdout: "public_network": "", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "public_network_interface": "", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "qat_compressor_enabled": "false", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "qat_compressor_session_max_number": "256", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "rados_mon_op_timeout": "0", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "rados_osd_op_timeout": "0", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "rados_tracing": "false", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_atime_update_interval": "60", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_balance_parent_reads": "false", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_balance_snap_reads": "false", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_blkin_trace_all": "false", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-20T02:50:32.491 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_cache": "true", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_cache_policy": "writearound", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_cache_size": "33554432", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_clone_copy_on_read": "false", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_compression_hint": "none", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_concurrent_management_ops": "10", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_default_clone_format": "auto", 2024-04-20T02:50:32.492 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_default_data_pool": "", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_default_features": "61", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_default_format": "2", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_default_map_options": "", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_default_order": "22", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_default_pool": "rbd", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_default_stripe_count": "0", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_default_stripe_unit": "0", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_enable_alloc_hint": "true", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-20T02:50:32.493 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_io_scheduler": "simple", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_order": "24", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_pool": "", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_journal_splay_width": "4", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_localize_parent_reads": "false", 2024-04-20T02:50:32.494 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_localize_snap_reads": "false", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-20T02:50:32.495 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-20T02:50:32.496 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_mtime_update_interval": "60", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_non_blocking_aio": "true", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_op_thread_timeout": "60", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_op_threads": "1", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_parent_cache_enabled": "false", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_plugins": "", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_bps_burst": "0", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_bps_limit": "0", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_exclude_ops": "0", 2024-04-20T02:50:32.497 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_iops_burst": "0", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_iops_limit": "0", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-20T02:50:32.498 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_read_from_replica_policy": "default", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_skip_partial_discard": "true", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_tracing": "false", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_validate_names": "true", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd_validate_pool": "true", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "restapi_base_url": "", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "restapi_log_level": "", 2024-04-20T02:50:32.499 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_acl_grants_max_num": "100", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_admin_entry": "admin", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_backend_store": "rados", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_barbican_url": "", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_beast_enable_async": "true", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_cache_enabled": "true", 2024-04-20T02:50:32.500 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_cache_expiry_interval": "900", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_cache_lru_size": "10000", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_config_store": "rados", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_content_length_compat": "false", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_copy_obj_progress": "true", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_cors_rules_max_num": "100", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_cross_domain_policy": "", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_kmip_addr": "", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_kmip_password": "", 2024-04-20T02:50:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_kmip_username": "", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_require_ssl": "true", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-20T02:50:32.502 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_vault_addr": "", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_vault_auth": "token", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_vault_namespace": "", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_vault_prefix": "", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_vault_token_file": "", 2024-04-20T02:50:32.503 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_curl_buffersize": "524288", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_curl_low_speed_time": "300", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-20T02:50:32.504 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_data_log_changes_size": "1000", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_data_log_num_shards": "128", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_data_log_window": "30", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-20T02:50:32.505 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dns_name": "", 2024-04-20T02:50:32.506 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dns_s3website_name": "", 2024-04-20T02:50:32.507 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_dynamic_resharding": "true", 2024-04-20T02:50:32.507 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-20T02:50:32.507 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_enable_gc_threads": "true", 2024-04-20T02:50:32.507 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_enable_lc_threads": "true", 2024-04-20T02:50:32.507 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_enable_ops_log": "false", 2024-04-20T02:50:32.507 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_enable_quota_threads": "true", 2024-04-20T02:50:32.507 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_enable_static_website": "false", 2024-04-20T02:50:32.507 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_enable_usage_log": "false", 2024-04-20T02:50:32.507 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_enforce_swift_acls": "true", 2024-04-20T02:50:32.507 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_exit_timeout_secs": "120", 2024-04-20T02:50:32.508 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_expose_bucket": "false", 2024-04-20T02:50:32.508 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_extended_http_attrs": "", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_filter": "none", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_frontends": "beast port=7480", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_gc_max_deferred": "50", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_gc_max_objs": "32", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_gc_processor_period": "3600", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-20T02:50:32.509 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_init_timeout": "300", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_admin_domain": "", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_admin_password": "", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_admin_password_path": "", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_admin_project": "", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_admin_tenant": "", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_admin_token": "", 2024-04-20T02:50:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_admin_token_path": "", 2024-04-20T02:50:32.511 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_admin_user": "", 2024-04-20T02:50:32.511 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_api_version": "2", 2024-04-20T02:50:32.511 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_barbican_domain": "", 2024-04-20T02:50:32.511 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_barbican_password": "", 2024-04-20T02:50:32.511 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_barbican_project": "", 2024-04-20T02:50:32.511 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-20T02:50:32.511 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_barbican_user": "", 2024-04-20T02:50:32.511 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-20T02:50:32.511 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-20T02:50:32.511 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_url": "", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_lc_debug_interval": "-1", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_lc_lock_max_time": "90", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_lc_max_objs": "32", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_lc_max_rules": "1000", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_lc_max_worker": "3", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_lc_thread_delay": "0", 2024-04-20T02:50:32.512 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-20T02:50:32.513 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ldap_dnattr": "uid", 2024-04-20T02:50:32.513 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-20T02:50:32.513 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ldap_searchfilter": "", 2024-04-20T02:50:32.513 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-20T02:50:32.513 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ldap_uri": "ldaps://", 2024-04-20T02:50:32.513 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-20T02:50:32.513 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-20T02:50:32.513 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-20T02:50:32.514 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_log_http_headers": "", 2024-04-20T02:50:32.514 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-20T02:50:32.514 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-20T02:50:32.514 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_log_object_name_utc": "false", 2024-04-20T02:50:32.514 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-04-20T02:50:32.515 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_attr_name_len": "0", 2024-04-20T02:50:32.515 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_attr_size": "0", 2024-04-20T02:50:32.515 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_chunk_size": "4194304", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_listing_results": "1000", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_notify_retries": "10", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_put_param_size": "1048576", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_put_size": "5368709120", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_max_slo_entries": "1000", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_md_log_max_shards": "64", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-20T02:50:32.516 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-20T02:50:32.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-20T02:50:32.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_mp_lock_max_time": "600", 2024-04-20T02:50:32.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-20T02:50:32.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-20T02:50:32.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-20T02:50:32.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-20T02:50:32.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-20T02:50:32.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-20T02:50:32.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-20T02:50:32.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-20T02:50:32.518 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_max_gc": "300", 2024-04-20T02:50:32.518 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-20T02:50:32.518 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-20T02:50:32.518 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-20T02:50:32.518 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-20T02:50:32.518 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-20T02:50:32.518 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-20T02:50:32.518 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-20T02:50:32.518 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_num_async_rados_threads": "32", 2024-04-20T02:50:32.518 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_num_control_oids": "8", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_numa_node": "-1", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_objexp_chunk_size": "100", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_objexp_gc_interval": "600", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_op_thread_timeout": "600", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_op_tracing": "false", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_opa_token": "", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_opa_url": "", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_opa_verify_ssl": "true", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-20T02:50:32.519 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ops_log_rados": "false", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_ops_log_socket_path": "", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_period_push_interval": "2.000000", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_print_continue": "true", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-20T02:50:32.520 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_rados_tracing": "false", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_realm": "", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_realm_id": "", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_region": "", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_request_uri": "", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_reshard_batch_size": "64", 2024-04-20T02:50:32.521 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_reshard_max_aio": "128", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_reshard_num_logs": "16", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_reshard_thread_interval": "600", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_resolve_cname": "false", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_run_sync_thread": "true", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-20T02:50:32.522 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_scheduler_type": "throttler", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_script_uri": "", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_service_provider_name": "", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sts_client_id": "", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sts_client_secret": "", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sts_entry": "sts", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sts_key": "sts", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sts_min_session_duration": "900", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sts_token_introspection_url": "", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_account_in_url": "false", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_auth_entry": "auth", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_auth_url": "", 2024-04-20T02:50:32.523 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_custom_header": "", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_need_stats": "true", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_tenant_name": "", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_token_expiration": "86400", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_url": "", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_url_prefix": "swift", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_lease_period": "120", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-20T02:50:32.524 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-20T02:50:32.525 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-20T02:50:32.525 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-20T02:50:32.525 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-20T02:50:32.525 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-20T02:50:32.525 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_thread_pool_size": "512", 2024-04-20T02:50:32.525 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_torrent_comment": "", 2024-04-20T02:50:32.525 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_torrent_createby": "", 2024-04-20T02:50:32.525 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_torrent_encoding": "", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_torrent_flag": "false", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_torrent_origin": "", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_torrent_tracker": "", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_trust_forwarded_https": "false", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_usage_max_shards": "32", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_usage_max_user_shards": "1", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_use_opa_authz": "false", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_user_max_buckets": "1000", 2024-04-20T02:50:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_user_policies_max_num": "100", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_user_unique_email": "true", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_verify_ssl": "true", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_zone": "", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_zone_id": "", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_zonegroup": "", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_zonegroup_id": "", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-20T02:50:32.527 INFO:teuthology.orchestra.run.smithi105.stdout: "rgwlc_auto_session_clear": "true", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_block_size": "4096", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_cache_size": "536870912", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-20T02:50:32.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_index_type": "binary_search", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_partition_filters": "false", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_perf": "false", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "run_dir": "/var/run/ceph", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_block_create": "true", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_cache_lru_size": "67108864", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-20T02:50:32.529 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_default_max_object_size": "16777216", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_device_size": "53687091200", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_journal_batch_capacity": "16", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_main_device_type": "SSD", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "seastore_segment_size": "67108864", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "set_keepcaps": "false", 2024-04-20T02:50:32.530 INFO:teuthology.orchestra.run.smithi105.stdout: "setgroup": "", 2024-04-20T02:50:32.531 INFO:teuthology.orchestra.run.smithi105.stdout: "setuser": "", 2024-04-20T02:50:32.531 INFO:teuthology.orchestra.run.smithi105.stdout: "setuser_match_path": "", 2024-04-20T02:50:32.531 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-20T02:50:32.531 INFO:teuthology.orchestra.run.smithi105.stdout: "thp": "false", 2024-04-20T02:50:32.531 INFO:teuthology.orchestra.run.smithi105.stdout: "threadpool_default_timeout": "60", 2024-04-20T02:50:32.531 INFO:teuthology.orchestra.run.smithi105.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-20T02:50:32.531 INFO:teuthology.orchestra.run.smithi105.stdout: "throttler_perf_counter": "true" 2024-04-20T02:50:32.532 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:32.534 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-20T02:50:32.669 INFO:teuthology.orchestra.run.smithi105.stdout:mon.a: {} 2024-04-20T02:50:32.681 INFO:teuthology.orchestra.run.smithi105.stdout:mon.b: {} 2024-04-20T02:50:32.690 INFO:teuthology.orchestra.run.smithi105.stdout:mon.c: {} 2024-04-20T02:50:32.691 INFO:teuthology.orchestra.run.smithi105.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T02:50:32.692 INFO:teuthology.orchestra.run.smithi105.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T02:50:32.692 INFO:teuthology.orchestra.run.smithi105.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T02:50:32.702 DEBUG:teuthology.orchestra.run.smithi138:> type ceph-objectstore-tool 2024-04-20T02:50:32.722 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-20T02:50:32.723 DEBUG:teuthology.orchestra.run.smithi105:> type ceph-objectstore-tool 2024-04-20T02:50:32.739 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-20T02:50:32.739 INFO:teuthology.run_tasks:Running task exec... 2024-04-20T02:50:32.748 INFO:teuthology.task.exec:Executing custom commands... 2024-04-20T02:50:32.748 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:50:32.748 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create low_tier 4' 2024-04-20T02:50:32.749 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-20T02:50:32.749 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-20T02:50:32.749 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:50:32.749 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-20T02:50:32.749 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-20T02:50:32.750 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:50:32.750 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-20T02:50:32.750 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:50:32.751 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-20T02:50:32.751 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:50:32.763 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:32.764+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:32.851 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:32.852+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:32.920 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:32.921 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:32.921 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:50:32.921 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:32.934 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:50:32.953 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:32.953+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:32.959 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:50:32.997 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:50:33.034 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:50:33.055 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:33.055+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:33.072 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:50:33.092 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:33.092 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:33.103 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:50:33.103 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:50:33.127 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:50:33.147 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:50:33.155 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:33.155+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:33.185 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:50:33.187 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:50:33.187 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:50:33.187 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:50:33.187 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:50:33.187 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:50:33.187 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:50:33.187 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:50:33.226 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'low_tier' created 2024-04-20T02:50:33.242 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:33.242 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:33.243 INFO:teuthology.run_tasks:Running task rados... 2024-04-20T02:50:33.253 INFO:tasks.rados:Beginning rados... 2024-04-20T02:50:33.253 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-20T02:50:33.303 INFO:tasks.rados:joining rados 2024-04-20T02:50:33.303 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-04-20T02:50:33.303 INFO:tasks.rados:starting run 0 out of 1 2024-04-20T02:50:33.303 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-20T02:50:33.303 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-20T02:50:33.305 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:50:33.306 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:33.306+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:33.405 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:33.406+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:33.479 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:33.479 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:33.492 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:50:33.537 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:33.537+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:33.607 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:33.608+0000 7f80e85c1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 43320) UID: 0 2024-04-20T02:50:33.623 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:33.623 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:33.634 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:50:33.708 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:33.709+0000 7f80e85c1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 43320) UID: 0 2024-04-20T02:50:33.803 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:33.803 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:33.809 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:33.810+0000 7f80e85c1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 43320) UID: 0 2024-04-20T02:50:33.813 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:50:33.910 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:33.911+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:33.941 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:33.941 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:33.952 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:50:34.012 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:34.012+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:34.080 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:34.080 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:34.090 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:50:34.113 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:34.114+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:34.214 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:34.214+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:34.236 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:50:34.238 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'unique_pool_0' created 2024-04-20T02:50:34.258 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-20T02:50:34.273 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:34.273 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:34.284 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:50:34.315 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:34.315+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:34.416 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:34.416+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:34.458 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:34.458 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:34.468 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:50:34.517 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:34.518+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:34.598 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:34.598 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:34.608 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:50:34.617 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:34.618+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:34.718 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:34.719+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:34.735 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:34.735 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:34.745 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:50:34.820 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:34.820+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:34.887 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:34.887 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:34.897 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:50:34.921 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:34.921+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:35.022 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:35.022+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:35.026 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:35.026 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:35.036 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:50:35.123 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:35.124+0000 7f80e85c1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 43320) UID: 0 2024-04-20T02:50:35.166 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:35.166 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:35.176 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:50:35.199 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:50:35.238 INFO:teuthology.orchestra.run.smithi105.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-20T02:50:35.259 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:35.260+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:35.325 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:35.325+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:35.349 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:35.349 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:35.359 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:50:35.360 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:50:35.378 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:50:35.396 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:50:35.416 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:50:35.435 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:50:35.444 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:35.445+0000 7f80e85c1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 43320) UID: 0 2024-04-20T02:50:35.454 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:50:35.474 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:50:35.493 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:50:35.494 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:50:35.495 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:50:35.495 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:50:35.495 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:50:35.495 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:50:35.495 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:50:35.495 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:50:35.495 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:50:35.528 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:35.528+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:35.533 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:35.534 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:35.534 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:35.534 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:35.534 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:35.544 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:50:35.629 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:35.629+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:35.714 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:35.714 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:50:35.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:50:35.714 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [] 2024-04-20T02:50:35.714 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:35.724 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:50:35.729 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:35.730+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:35.830 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:35.830+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:35.851 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:35.852 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:35.852 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:35.852 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:35.862 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:50:35.931 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:35.931+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:36.030 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:36.030 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:36.030 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:36.030 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:36.030 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:36.031 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:36.032+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:36.040 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:50:36.132 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:36.133+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:35 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'21, mlcod=16'15)", 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.964670+0000", 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.24466151, 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054852659999999999, 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.210 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964670+0000", 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964670+0000", 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964670+0000", 2024-04-20T02:50:36.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964691+0000", 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1175e-05 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964698+0000", 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3950000000000003e-06 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964704+0000", 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4829999999999999e-06 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964747+0000", 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3016000000000003e-05 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964780+0000", 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3000999999999998e-05 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.970129+0000", 2024-04-20T02:50:36.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005349286 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.970156+0000", 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.639e-05 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:106 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'68, mlcod=16'64)", 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.136358+0000", 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.072973950000002, 2024-04-20T02:50:36.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049723069999999996, 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136358+0000", 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136358+0000", 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136357+0000", 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136379+0000", 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1742999999999998e-05 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136388+0000", 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.268e-06 2024-04-20T02:50:36.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136394+0000", 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7089999999999999e-06 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136424+0000", 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0052e-05 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136436+0000", 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1754e-05 2024-04-20T02:50:36.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141295+0000", 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004858996 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141330+0000", 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5383999999999999e-05 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.218 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:111 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'73, mlcod=16'65)", 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.138432+0000", 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.070899318999999, 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054900060000000004, 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138432+0000", 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138432+0000", 2024-04-20T02:50:36.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138432+0000", 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138438+0000", 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1909999999999997e-06 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138448+0000", 2024-04-20T02:50:36.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9660000000000008e-06 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138454+0000", 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4119999999999998e-06 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138504+0000", 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9301000000000001e-05 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138524+0000", 2024-04-20T02:50:36.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0319e-05 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.143889+0000", 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053652079999999998 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.143922+0000", 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3229000000000003e-05 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:112 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'74, mlcod=16'65)", 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.138900+0000", 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.070431558999999, 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050438439999999996, 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138900+0000", 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138900+0000", 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138900+0000", 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138948+0000", 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8612999999999997e-05 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138957+0000", 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5080000000000005e-06 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138962+0000", 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8099999999999997e-06 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138994+0000", 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2697999999999998e-05 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.139013+0000", 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8808e-05 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.143925+0000", 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049121440000000002 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.143944+0000", 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8853000000000001e-05 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.149068+0000", 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.060263995, 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050287819999999999, 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149068+0000", 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149068+0000", 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149067+0000", 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149113+0000", 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6001999999999998e-05 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149122+0000", 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2329999999999995e-06 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149128+0000", 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.339e-06 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149170+0000", 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2088000000000001e-05 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149182+0000", 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2653000000000001e-05 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.154064+0000", 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048819750000000002 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.154097+0000", 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2067000000000003e-05 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:131 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'89, mlcod=16'85)", 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.476200+0000", 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.733131733, 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099670190000000006, 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476200+0000", 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476200+0000", 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476199+0000", 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476232+0000", 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2197999999999999e-05 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476239+0000", 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6389999999999992e-06 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476249+0000", 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7310000000000005e-06 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476280+0000", 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1417000000000001e-05 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476298+0000", 2024-04-20T02:50:36.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7683999999999999e-05 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486147+0000", 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0098485159999999999 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486167+0000", 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0373e-05 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:132 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'90, mlcod=16'85)", 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.478573+0000", 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.730758502, 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0076055539999999996, 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478573+0000", 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478573+0000", 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478573+0000", 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478577+0000", 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7090000000000001e-06 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478585+0000", 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7100000000000007e-06 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478591+0000", 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2500000000000003e-06 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478656+0000", 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4111999999999999e-05 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478672+0000", 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6854000000000002e-05 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486169+0000", 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074966260000000002 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486179+0000", 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7520000000000008e-06 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.238 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:133 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'86)", 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.479193+0000", 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.730138760999999, 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069972599999999999, 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479193+0000", 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479193+0000", 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479192+0000", 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479197+0000", 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7829999999999998e-06 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479205+0000", 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7919999999999993e-06 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479210+0000", 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8570000000000003e-06 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479238+0000", 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8501e-05 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479253+0000", 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4483e-05 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486180+0000", 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069272320000000002 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486190+0000", 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0127e-05 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.242 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:134 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'86)", 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.479644+0000", 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.729687670000001, 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0065671790000000002, 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479644+0000", 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479644+0000", 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479643+0000", 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479669+0000", 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5230000000000001e-05 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479678+0000", 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4930000000000004e-06 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479685+0000", 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5220000000000004e-06 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479727+0000", 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2583000000000002e-05 2024-04-20T02:50:36.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479740+0000", 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.325e-05 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486192+0000", 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0064515759999999997 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486211+0000", 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9293e-05 2024-04-20T02:50:36.246 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:135 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'86)", 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.480066+0000", 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.729265671, 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.006170343, 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480066+0000", 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480066+0000", 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480066+0000", 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480070+0000", 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7110000000000003e-06 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480078+0000", 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7379999999999994e-06 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480083+0000", 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3210000000000004e-06 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480110+0000", 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6489999999999999e-05 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480131+0000", 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1039e-05 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486216+0000", 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060846099999999998 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486236+0000", 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0832999999999999e-05 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.250 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:136 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'86)", 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.480471+0000", 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.728860944999999, 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057863059999999997, 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480471+0000", 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480471+0000", 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480470+0000", 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480478+0000", 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.011e-06 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480488+0000", 2024-04-20T02:50:36.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3789999999999995e-06 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480494+0000", 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0850000000000002e-06 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480525+0000", 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1164999999999997e-05 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480540+0000", 2024-04-20T02:50:36.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4946999999999999e-05 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486240+0000", 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0056999360000000001 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486257+0000", 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7326999999999999e-05 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:137 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'95, mlcod=16'86)", 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.480922+0000", 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.728410056, 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053589800000000002, 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480922+0000", 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480922+0000", 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480921+0000", 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480943+0000", 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1934999999999999e-05 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480951+0000", 2024-04-20T02:50:36.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9699999999999999e-06 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480956+0000", 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9119999999999997e-06 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480993+0000", 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6956999999999999e-05 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481012+0000", 2024-04-20T02:50:36.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8607e-05 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486259+0000", 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005247425 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486281+0000", 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1713e-05 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:138 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'96, mlcod=16'86)", 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.481709+0000", 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.727623231000001, 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049116979999999999, 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481709+0000", 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481709+0000", 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481708+0000", 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481712+0000", 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3420000000000001e-06 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481719+0000", 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0469999999999996e-06 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481724+0000", 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2520000000000001e-06 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483637+0000", 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001912789 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483649+0000", 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2771e-05 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486578+0000", 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029286820000000002 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486620+0000", 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2224000000000001e-05 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:140 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'98, mlcod=16'87)", 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.482489+0000", 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.726842484000001, 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066787219999999998, 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482489+0000", 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482489+0000", 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482488+0000", 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482507+0000", 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9338999999999999e-05 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482521+0000", 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482527+0000", 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7610000000000004e-06 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484064+0000", 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001536737 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484086+0000", 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2178999999999999e-05 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489141+0000", 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050549540000000004 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489168+0000", 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7022e-05 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.266 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:141 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'87)", 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.483045+0000", 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.726286463999999, 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0061406789999999996, 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483045+0000", 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483045+0000", 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483045+0000", 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483050+0000", 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3310000000000003e-06 2024-04-20T02:50:36.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483059+0000", 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9230000000000008e-06 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483066+0000", 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3250000000000002e-06 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486305+0000", 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0032387090000000002 2024-04-20T02:50:36.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486325+0000", 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9905e-05 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489170+0000", 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0028451230000000002 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489186+0000", 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5883e-05 2024-04-20T02:50:36.270 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:155 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'109, mlcod=16'105)", 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.952241+0000", 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.257090420000001, 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0056084400000000001, 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952241+0000", 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952241+0000", 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952241+0000", 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952298+0000", 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7089999999999997e-05 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952307+0000", 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7609999999999996e-06 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952314+0000", 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9999999999999999e-06 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952348+0000", 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3797999999999997e-05 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952364+0000", 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6010000000000001e-05 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957821+0000", 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005457249 2024-04-20T02:50:36.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957850+0000", 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8969999999999999e-05 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:156 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'110, mlcod=16'105)", 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.952994+0000", 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.256338076, 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0070517490000000004, 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.275 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952994+0000", 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952994+0000", 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952993+0000", 2024-04-20T02:50:36.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953005+0000", 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1412e-05 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953013+0000", 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0129999999999993e-06 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953019+0000", 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5760000000000002e-06 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.954938+0000", 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019186170000000001 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.954959+0000", 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0721e-05 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960017+0000", 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00505843 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960045+0000", 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8450999999999999e-05 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:157 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'111, mlcod=16'105)", 2024-04-20T02:50:36.279 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.953402+0000", 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.255929388, 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066620789999999996, 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953402+0000", 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953402+0000", 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953402+0000", 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953407+0000", 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1440000000000002e-06 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953413+0000", 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.981e-06 2024-04-20T02:50:36.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953421+0000", 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6199999999999999e-06 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955173+0000", 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001752094 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955192+0000", 2024-04-20T02:50:36.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9570000000000001e-05 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960047+0000", 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048545230000000003 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960064+0000", 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7533e-05 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=16'105)", 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.953833+0000", 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.255498629, 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0065076650000000002, 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953833+0000", 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953833+0000", 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953833+0000", 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953861+0000", 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8524999999999999e-05 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953867+0000", 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6500000000000001e-06 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953871+0000", 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0910000000000003e-06 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955472+0000", 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016007300000000001 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955495+0000", 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.317e-05 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960316+0000", 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048213479999999996 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960341+0000", 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4409999999999998e-05 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=16'105)", 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.955527+0000", 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.253804782, 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004827904, 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955527+0000", 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955527+0000", 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955526+0000", 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955534+0000", 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6680000000000001e-06 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955545+0000", 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.149e-05 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955553+0000", 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6459999999999993e-06 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957592+0000", 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020391799999999998 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957613+0000", 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1053e-05 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960343+0000", 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002729469 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960355+0000", 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2055000000000001e-05 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.291 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.292 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.292 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.292 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:36.292 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:50:36.293 DEBUG:teuthology.orchestra.run.smithi105:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0 2024-04-20T02:50:36.294 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:36.293+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:36.295 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:50:36.326 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight read -> 100 2024-04-20T02:50:36.326 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight write -> 25 2024-04-20T02:50:36.326 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight delete -> 10 2024-04-20T02:50:36.326 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight snap_create -> 10 2024-04-20T02:50:36.326 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight snap_remove -> 10 2024-04-20T02:50:36.326 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight rollback -> 10 2024-04-20T02:50:36.326 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight set_chunk -> 30 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight tier_promote -> 10 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight tier_evict -> 10 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight tier_flush -> 5 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout:adding op weight write_excl -> 25 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout:ceph version 18.2.2-1252-g15eecde2 (15eecde2bab9803889d929c7e423d07970d8cbed) reef (stable) 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout:Configuration: 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout: Number of operations: 1500 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout: Number of objects: 50 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout: Max in flight operations: 16 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout: Object size (in bytes): 4000000 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout: Write stride min: 400000 2024-04-20T02:50:36.327 INFO:tasks.rados.rados.0.smithi105.stdout: Write stride max: 800000 2024-04-20T02:50:36.394 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:36.395+0000 7f80e85c1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 43320) UID: 0 2024-04-20T02:50:36.466 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:36.466 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:36.466 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:36.466 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:36.476 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:50:36.518 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:36.518+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:36.596 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:36.597+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:36.644 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:36.644 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:36.644 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:36.644 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:36.644 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:36.654 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:50:36.697 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:36.698+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:36.798 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:36.798+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.219803884000001, 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.825 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:50:36.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.135845+0000", 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.687873642, 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0091040970000000002, 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.832 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 114 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135845+0000", 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135845+0000", 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135844+0000", 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135848+0000", 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2230000000000001e-06 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135854+0000", 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5679999999999999e-06 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135858+0000", 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1099999999999996e-06 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141165+0000", 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053073260000000002 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141221+0000", 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5161000000000003e-05 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141303+0000", 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.2003999999999996e-05 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144686+0000", 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033838290000000001 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144857+0000", 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017099800000000001 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144906+0000", 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8812e-05 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144930+0000", 2024-04-20T02:50:36.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4096000000000001e-05 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144949+0000", 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8479999999999999e-05 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.135864+0000", 2024-04-20T02:50:36.838 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.68785488, 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0091993409999999998, 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 115 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135864+0000", 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135864+0000", 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135863+0000", 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135918+0000", 2024-04-20T02:50:36.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.452e-05 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135925+0000", 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4410000000000004e-06 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135931+0000", 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8540000000000002e-06 2024-04-20T02:50:36.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141573+0000", 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0056424960000000003 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141629+0000", 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5260999999999998e-05 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141742+0000", 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000112898 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144862+0000", 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003120795 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144889+0000", 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6401e-05 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144992+0000", 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103689 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145032+0000", 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9167e-05 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145063+0000", 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1251999999999998e-05 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.844 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.135939+0000", 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.687779637999999, 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010214235, 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.845 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 116 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135939+0000", 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135939+0000", 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135938+0000", 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135943+0000", 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4159999999999997e-06 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135949+0000", 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2009999999999996e-06 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135954+0000", 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2320000000000001e-06 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145095+0000", 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.009141415 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145142+0000", 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6573000000000003e-05 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145224+0000", 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.2261000000000004e-05 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145863+0000", 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063818000000000002 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146001+0000", 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013806300000000001 2024-04-20T02:50:36.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146084+0000", 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3651000000000003e-05 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146118+0000", 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3442000000000003e-05 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146153+0000", 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5345000000000001e-05 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.135960+0000", 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.687758127999999, 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010370449, 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.851 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 117 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135960+0000", 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135960+0000", 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135960+0000", 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135966+0000", 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9529999999999996e-06 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135972+0000", 2024-04-20T02:50:36.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135978+0000", 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5060000000000003e-06 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145515+0000", 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095370119999999992 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145562+0000", 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145646+0000", 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3327999999999999e-05 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146255+0000", 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00060893600000000003 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146263+0000", 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5429999999999994e-06 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146285+0000", 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1458999999999999e-05 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146311+0000", 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6526e-05 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146331+0000", 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9678e-05 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:131 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475068+0000", 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.348650895999999, 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011277362000000001, 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.857 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 131 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475068+0000", 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475068+0000", 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475067+0000", 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475095+0000", 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8326e-05 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475104+0000", 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6729999999999998e-06 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475108+0000", 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.529e-06 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475874+0000", 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076629400000000005 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475955+0000", 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0963000000000004e-05 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476098+0000", 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014343099999999999 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478942+0000", 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0028435259999999999 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481887+0000", 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029456679999999998 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486272+0000", 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004384125 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486308+0000", 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6786000000000003e-05 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486345+0000", 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6461000000000002e-05 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475113+0000", 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.348604941000001, 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011336222, 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.863 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 132 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475113+0000", 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475113+0000", 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475113+0000", 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475115+0000", 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2249999999999999e-06 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475119+0000", 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3289999999999998e-06 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475122+0000", 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9239999999999999e-06 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478386+0000", 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0032647829999999998 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478437+0000", 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0068000000000002e-05 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478510+0000", 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3889999999999999e-05 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481487+0000", 2024-04-20T02:50:36.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002976472 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481891+0000", 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040387700000000002 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486399+0000", 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004508359 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486428+0000", 2024-04-20T02:50:36.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8852000000000001e-05 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486450+0000", 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1685999999999999e-05 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475313+0000", 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.348405353, 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011242378000000001, 2024-04-20T02:50:36.869 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 133 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475313+0000", 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475313+0000", 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475312+0000", 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475319+0000", 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8589999999999999e-06 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475330+0000", 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1209000000000001e-05 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475338+0000", 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8560000000000007e-06 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479010+0000", 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003672219 2024-04-20T02:50:36.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479057+0000", 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6684000000000002e-05 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479136+0000", 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8978000000000002e-05 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481894+0000", 2024-04-20T02:50:36.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0027578780000000001 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483367+0000", 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014729750000000001 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486484+0000", 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0031168210000000001 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486522+0000", 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7936999999999999e-05 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486555+0000", 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3782000000000003e-05 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475361+0000", 2024-04-20T02:50:36.875 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.348357553, 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011315524, 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 134 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475361+0000", 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475361+0000", 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475360+0000", 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475403+0000", 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2311000000000002e-05 2024-04-20T02:50:36.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475410+0000", 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475416+0000", 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3219999999999998e-06 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479420+0000", 2024-04-20T02:50:36.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0040038000000000001 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479471+0000", 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1582999999999997e-05 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479548+0000", 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6747000000000002e-05 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481897+0000", 2024-04-20T02:50:36.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0023488390000000001 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483386+0000", 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014890719999999999 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486610+0000", 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0032237810000000002 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486656+0000", 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5940999999999999e-05 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486676+0000", 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0608999999999999e-05 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475426+0000", 2024-04-20T02:50:36.881 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.348292167, 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011332076, 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 135 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475426+0000", 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475426+0000", 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475426+0000", 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475430+0000", 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5639999999999998e-06 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475436+0000", 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7799999999999997e-06 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475442+0000", 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479825+0000", 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0043834360000000001 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479908+0000", 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3195000000000006e-05 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480007+0000", 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8997000000000001e-05 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481900+0000", 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0018928529999999999 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483406+0000", 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001505981 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486710+0000", 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033038949999999998 2024-04-20T02:50:36.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486739+0000", 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9464000000000001e-05 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486758+0000", 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8969000000000001e-05 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475447+0000", 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.348271212, 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011384475, 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 136 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475447+0000", 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475447+0000", 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475447+0000", 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475453+0000", 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8409999999999996e-06 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475461+0000", 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.2109999999999998e-06 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475466+0000", 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4809999999999997e-06 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480251+0000", 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0047850510000000002 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480302+0000", 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0801999999999999e-05 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480377+0000", 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4772999999999997e-05 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481903+0000", 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001526052 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483768+0000", 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0018654089999999999 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486792+0000", 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003023141 2024-04-20T02:50:36.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486814+0000", 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1942000000000001e-05 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486832+0000", 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8117e-05 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475472+0000", 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.348246227000001, 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011424245, 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475472+0000", 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475472+0000", 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475472+0000", 2024-04-20T02:50:36.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475539+0000", 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6761999999999995e-05 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475545+0000", 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5899999999999996e-06 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475551+0000", 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7760000000000003e-06 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480608+0000", 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050569439999999998 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480692+0000", 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3689000000000005e-05 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480823+0000", 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013148399999999999 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483820+0000", 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029966509999999999 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484012+0000", 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019240199999999999 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486858+0000", 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002845991 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486880+0000", 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1503000000000001e-05 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486896+0000", 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6891000000000001e-05 2024-04-20T02:50:36.899 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475557+0000", 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.348161257000001, 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011424037999999999, 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.900 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 138 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475557+0000", 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475557+0000", 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475557+0000", 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475561+0000", 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8890000000000004e-06 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475574+0000", 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3006999999999999e-05 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475584+0000", 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.027e-05 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481511+0000", 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0059274560000000002 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481576+0000", 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4918000000000004e-05 2024-04-20T02:50:36.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481650+0000", 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3668e-05 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483879+0000", 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00222923 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484022+0000", 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014319100000000001 2024-04-20T02:50:36.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486943+0000", 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029209969999999998 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486964+0000", 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0843999999999999e-05 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486981+0000", 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7076e-05 2024-04-20T02:50:36.905 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475593+0000", 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.34812514, 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011444096000000001, 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.906 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 139 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475593+0000", 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475593+0000", 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475593+0000", 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475597+0000", 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2080000000000002e-06 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475602+0000", 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.271e-06 2024-04-20T02:50:36.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475606+0000", 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.517e-06 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481928+0000", 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.006322449 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481976+0000", 2024-04-20T02:50:36.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7840999999999998e-05 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482058+0000", 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.1842999999999996e-05 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483920+0000", 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0018621149999999999 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484027+0000", 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106959 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.487000+0000", 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029727669999999999 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.487019+0000", 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9542999999999999e-05 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.487037+0000", 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8107000000000001e-05 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475613+0000", 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.348104996, 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013679606, 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.912 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 140 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475613+0000", 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475613+0000", 2024-04-20T02:50:36.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475613+0000", 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475665+0000", 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1641999999999998e-05 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475672+0000", 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5100000000000001e-06 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475677+0000", 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4290000000000003e-06 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482302+0000", 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066251310000000002 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.915 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482350+0000", 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482422+0000", 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2260000000000003e-05 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483956+0000", 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015343589999999999 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484032+0000", 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5877000000000006e-05 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489249+0000", 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0052166499999999998 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489275+0000", 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6148000000000001e-05 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489293+0000", 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7961999999999998e-05 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475681+0000", 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.348037457, 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013671255, 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.918 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 141 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475681+0000", 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475681+0000", 2024-04-20T02:50:36.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475681+0000", 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475683+0000", 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4569999999999999e-06 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475686+0000", 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0599999999999999e-06 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475689+0000", 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.909e-06 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482873+0000", 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0071836540000000003 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482921+0000", 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8637999999999997e-05 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482995+0000", 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769000000000005e-05 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484038+0000", 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010431349999999999 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486921+0000", 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0028831360000000001 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489314+0000", 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002392936 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489336+0000", 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1648e-05 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489352+0000", 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6240999999999999e-05 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:156 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.950943+0000", 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.872775628999999, 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092894120000000004, 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.924 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 156 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950943+0000", 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950943+0000", 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950942+0000", 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950953+0000", 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0438999999999999e-05 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950956+0000", 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.269e-06 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950960+0000", 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7290000000000002e-06 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952711+0000", 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017508910000000001 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952797+0000", 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6635e-05 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952926+0000", 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000128746 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957448+0000", 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004521466 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.958448+0000", 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00099999799999999999 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960150+0000", 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001702582 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960196+0000", 2024-04-20T02:50:36.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5766000000000002e-05 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960232+0000", 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6179e-05 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.950963+0000", 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.8727555, 2024-04-20T02:50:36.930 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0093661939999999996, 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 157 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950963+0000", 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950963+0000", 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950963+0000", 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950965+0000", 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9690000000000001e-06 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950967+0000", 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7049999999999999e-06 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950970+0000", 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7300000000000001e-06 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953211+0000", 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022404759999999999 2024-04-20T02:50:36.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953259+0000", 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8223000000000002e-05 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953338+0000", 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8949000000000006e-05 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957455+0000", 2024-04-20T02:50:36.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041168819999999997 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.958468+0000", 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010132170000000001 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960266+0000", 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017985099999999999 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960295+0000", 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8753000000000001e-05 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960329+0000", 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4035000000000002e-05 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.950979+0000", 2024-04-20T02:50:36.936 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.87273905, 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0094873709999999997, 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 158 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950979+0000", 2024-04-20T02:50:36.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950979+0000", 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950979+0000", 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951015+0000", 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6418999999999998e-05 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951020+0000", 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0600000000000001e-06 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951023+0000", 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7639999999999999e-06 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953612+0000", 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002588595 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953661+0000", 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8893000000000001e-05 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953739+0000", 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7818999999999995e-05 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.958484+0000", 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0047458129999999998 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.959996+0000", 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015118670000000001 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960409+0000", 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00041247799999999998 2024-04-20T02:50:36.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960432+0000", 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3620000000000001e-05 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960467+0000", 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4372999999999998e-05 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.951027+0000", 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.872691816, 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010295645000000001, 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 159 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951027+0000", 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951027+0000", 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951026+0000", 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951030+0000", 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2820000000000001e-06 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951032+0000", 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6979999999999998e-06 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951035+0000", 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8100000000000002e-06 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955300+0000", 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0042653589999999998 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955348+0000", 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7414999999999999e-05 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955427+0000", 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8737000000000002e-05 2024-04-20T02:50:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960000+0000", 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004573293 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960504+0000", 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050367299999999995 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.961266+0000", 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076279099999999997 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.961288+0000", 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1814000000000002e-05 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.961322+0000", 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4115000000000001e-05 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:36.948 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:36.950 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:36.951+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:36.950 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:50:37.050 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:37.050+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:37.079 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:37.079 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:37.079 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:50:37.079 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:37.089 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:50:37.173 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:37.173+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:37.252 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:37.252+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:37.252 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:50:37.267 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:37.267 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:37.268 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:50:37.268 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:50:37.268 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:37.278 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:50:37.353 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:37.354+0000 7f80e85c1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 43320) UID: 0 2024-04-20T02:50:37.449 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:37.449 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:50:37.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:50:37.449 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:50:37.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.449 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-04-20T02:50:37.449 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.924154+0000", 2024-04-20T02:50:37.449 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.524536924, 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003869146, 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924154+0000", 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924154+0000", 2024-04-20T02:50:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924153+0000", 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924159+0000", 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.161e-06 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924172+0000", 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2300000000000001e-05 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924182+0000", 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0332999999999999e-05 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924218+0000", 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5800000000000003e-05 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924245+0000", 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6954e-05 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.927990+0000", 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003745253 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.928023+0000", 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2685000000000003e-05 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:33 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'19, mlcod=16'14)", 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.963561+0000", 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.485129671999999, 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040083649999999998, 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963561+0000", 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963561+0000", 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963560+0000", 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963567+0000", 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3609999999999997e-06 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963576+0000", 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4669999999999993e-06 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963583+0000", 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3250000000000002e-06 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963635+0000", 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1597000000000001e-05 2024-04-20T02:50:37.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963654+0000", 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9531999999999999e-05 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967545+0000", 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0038906840000000002 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967569+0000", 2024-04-20T02:50:37.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4108e-05 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:34 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'20, mlcod=16'15)", 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.964277+0000", 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.484413839999998, 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033064570000000001, 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964277+0000", 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964277+0000", 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964276+0000", 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964282+0000", 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9800000000000003e-06 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964291+0000", 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9029999999999993e-06 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964296+0000", 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.101e-06 2024-04-20T02:50:37.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964322+0000", 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5647e-05 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964336+0000", 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4452e-05 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967573+0000", 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003236591 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967583+0000", 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0429e-05 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:35 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'21, mlcod=16'15)", 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.964635+0000", 2024-04-20T02:50:37.462 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.484055679000001, 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037872499999999998, 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964635+0000", 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964635+0000", 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964634+0000", 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964689+0000", 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4784999999999998e-05 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964696+0000", 2024-04-20T02:50:37.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9870000000000002e-06 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964700+0000", 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.031e-06 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967084+0000", 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0023832760000000001 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967131+0000", 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7444999999999997e-05 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968377+0000", 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001246055 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968422+0000", 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5089000000000001e-05 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:36 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'22, mlcod=16'16)", 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.965393+0000", 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.483297670999999, 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0035673390000000001, 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965393+0000", 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965393+0000", 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965392+0000", 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965411+0000", 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8612000000000001e-05 2024-04-20T02:50:37.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965417+0000", 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.913e-06 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965421+0000", 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8630000000000001e-06 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967375+0000", 2024-04-20T02:50:37.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019539560000000002 2024-04-20T02:50:37.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967389+0000", 2024-04-20T02:50:37.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4496e-05 2024-04-20T02:50:37.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968940+0000", 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015508939999999999 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968960+0000", 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0037000000000001e-05 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.471 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'16)", 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.965893+0000", 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.482797852000001, 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030820349999999999, 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965893+0000", 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965893+0000", 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965892+0000", 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965896+0000", 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.16e-06 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965904+0000", 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0879999999999997e-06 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965907+0000", 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.574e-06 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967603+0000", 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001696078 2024-04-20T02:50:37.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967618+0000", 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4718e-05 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968962+0000", 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001344095 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968975+0000", 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734999999999999e-05 2024-04-20T02:50:37.475 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'66)", 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.141789+0000", 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.306901738000001, 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0029188780000000002, 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141789+0000", 2024-04-20T02:50:37.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141789+0000", 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141789+0000", 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141861+0000", 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2187999999999994e-05 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141866+0000", 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4700000000000001e-06 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141870+0000", 2024-04-20T02:50:37.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1420000000000003e-06 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.142829+0000", 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00095890999999999999 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.142842+0000", 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2576e-05 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.144690+0000", 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001847841 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.144708+0000", 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7986999999999999e-05 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:133 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'86)", 2024-04-20T02:50:37.480 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.479206+0000", 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.969485036999998, 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040552940000000001, 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479206+0000", 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479206+0000", 2024-04-20T02:50:37.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479205+0000", 2024-04-20T02:50:37.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479212+0000", 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1590000000000004e-06 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479222+0000", 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5920000000000007e-06 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479227+0000", 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2730000000000002e-06 2024-04-20T02:50:37.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479250+0000", 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2872999999999999e-05 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479261+0000", 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0818999999999999e-05 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483242+0000", 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039811780000000001 2024-04-20T02:50:37.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483261+0000", 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9018e-05 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:134 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'86)", 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.479634+0000", 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.969056516999999, 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0036374179999999999, 2024-04-20T02:50:37.485 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479634+0000", 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479634+0000", 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479633+0000", 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479671+0000", 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7889000000000003e-05 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479697+0000", 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5454999999999999e-05 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479702+0000", 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8010000000000003e-06 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479741+0000", 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9075000000000001e-05 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479751+0000", 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0661e-05 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483262+0000", 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003510614 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483271+0000", 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5519999999999993e-06 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:135 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'86)", 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.480079+0000", 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.968611816999999, 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0032009180000000001, 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480079+0000", 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480079+0000", 2024-04-20T02:50:37.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480078+0000", 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480083+0000", 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2979999999999999e-06 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480096+0000", 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2926e-05 2024-04-20T02:50:37.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480102+0000", 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.31e-06 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480124+0000", 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2220000000000001e-05 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480135+0000", 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1164e-05 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483272+0000", 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031368989999999999 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483280+0000", 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7190000000000001e-06 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:136 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'86)", 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.480416+0000", 2024-04-20T02:50:37.493 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.968274503, 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003224785, 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480416+0000", 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480416+0000", 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480416+0000", 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480419+0000", 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1709999999999998e-06 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480424+0000", 2024-04-20T02:50:37.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8679999999999998e-06 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480427+0000", 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6799999999999999e-06 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480450+0000", 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3042000000000001e-05 2024-04-20T02:50:37.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480463+0000", 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2377e-05 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483628+0000", 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031652289999999999 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483641+0000", 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2764e-05 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:141 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'87)", 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.483058+0000", 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.965632927000001, 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031466419999999998, 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.498 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483058+0000", 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483058+0000", 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483057+0000", 2024-04-20T02:50:37.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483062+0000", 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1019999999999996e-06 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483070+0000", 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5669999999999997e-06 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483074+0000", 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7369999999999997e-06 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483108+0000", 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3399999999999999e-05 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483119+0000", 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0756e-05 2024-04-20T02:50:37.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.486176+0000", 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030574320000000001 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.486204+0000", 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8229e-05 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:144 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'102, mlcod=16'98)", 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.491197+0000", 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.957493267, 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0055184850000000001, 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491197+0000", 2024-04-20T02:50:37.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491197+0000", 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491197+0000", 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491262+0000", 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4984e-05 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491269+0000", 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9310000000000002e-06 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491273+0000", 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8820000000000003e-06 2024-04-20T02:50:37.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491296+0000", 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3238999999999999e-05 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491309+0000", 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2729e-05 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.496690+0000", 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053810489999999997 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.496716+0000", 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6069999999999999e-05 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:145 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'103, mlcod=16'98)", 2024-04-20T02:50:37.507 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.491959+0000", 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.956731929, 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0047703800000000003, 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491959+0000", 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491959+0000", 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491958+0000", 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.492035+0000", 2024-04-20T02:50:37.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6979999999999998e-05 2024-04-20T02:50:37.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.492041+0000", 2024-04-20T02:50:37.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1340000000000001e-06 2024-04-20T02:50:37.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.492045+0000", 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0049999999999998e-06 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.492070+0000", 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4272000000000001e-05 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.492083+0000", 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3791999999999999e-05 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.496717+0000", 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046339049999999998 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.496729+0000", 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1637999999999999e-05 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.512 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:146 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'104, mlcod=16'102)", 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.497330+0000", 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.951360438000002, 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0035693740000000002, 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497330+0000", 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497330+0000", 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497329+0000", 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497343+0000", 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3930999999999999e-05 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497353+0000", 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5659999999999996e-06 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497358+0000", 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0340000000000004e-06 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497391+0000", 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2957999999999999e-05 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497405+0000", 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3991e-05 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.500872+0000", 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034673960000000002 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.500899+0000", 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.711e-05 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.516 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:155 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'109, mlcod=16'105)", 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.952207+0000", 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.496483168000001, 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0035612809999999999, 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952207+0000", 2024-04-20T02:50:37.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952207+0000", 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952207+0000", 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952315+0000", 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108133 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952322+0000", 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1080000000000004e-06 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952327+0000", 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3019999999999996e-06 2024-04-20T02:50:37.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952350+0000", 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3011e-05 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952364+0000", 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4046000000000001e-05 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955738+0000", 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033740179999999999 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955769+0000", 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1078000000000003e-05 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:156 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'110, mlcod=16'105)", 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.952957+0000", 2024-04-20T02:50:37.521 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.495733421000001, 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053524920000000004, 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952957+0000", 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952957+0000", 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952957+0000", 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952960+0000", 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3730000000000001e-06 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952966+0000", 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9120000000000003e-06 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.523 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952970+0000", 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6990000000000001e-06 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952996+0000", 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6084e-05 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953008+0000", 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2625e-05 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958283+0000", 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052744899999999997 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958310+0000", 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6693e-05 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.525 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:157 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'111, mlcod=16'105)", 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.953378+0000", 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.495312293000001, 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0049499729999999999, 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953378+0000", 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953378+0000", 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953378+0000", 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953383+0000", 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8509999999999998e-06 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953388+0000", 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5339999999999999e-06 2024-04-20T02:50:37.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953391+0000", 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9869999999999999e-06 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953416+0000", 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4312e-05 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953429+0000", 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3550000000000001e-05 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958312+0000", 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0048824610000000003 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958328+0000", 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6578999999999998e-05 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=16'105)", 2024-04-20T02:50:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.953823+0000", 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.494867279000001, 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0045220779999999997, 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953823+0000", 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953823+0000", 2024-04-20T02:50:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953823+0000", 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953874+0000", 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0890999999999998e-05 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953879+0000", 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7370000000000003e-06 2024-04-20T02:50:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953884+0000", 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0339999999999998e-06 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953907+0000", 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3059000000000001e-05 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953921+0000", 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3985000000000001e-05 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958329+0000", 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0044084939999999998 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958345+0000", 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6277999999999999e-05 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=16'105)", 2024-04-20T02:50:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.955539+0000", 2024-04-20T02:50:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.493151681000001, 2024-04-20T02:50:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056231900000000001, 2024-04-20T02:50:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955539+0000", 2024-04-20T02:50:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955539+0000", 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955538+0000", 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955544+0000", 2024-04-20T02:50:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4779999999999999e-06 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955554+0000", 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.815e-06 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955561+0000", 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6000000000000001e-06 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955594+0000", 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2838e-05 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955606+0000", 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1843e-05 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.961130+0000", 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0055239019999999998 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.961162+0000", 2024-04-20T02:50:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2335000000000001e-05 2024-04-20T02:50:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:37.538 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:37.538 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:50:37.540 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:37.540+0000 7f80e85c1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 43320) UID: 0 2024-04-20T02:50:37.540 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:50:37.640 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:37.640+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:37.712 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:37.713 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:37.713 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:50:37.713 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:37.723 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:50:37.727 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:50:37.741 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:37.742+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:37.747 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:50:37.750 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-20T02:50:37.750 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:50:37.751 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-04-20T02:50:37.751 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-20T02:50:37.751 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-20T02:50:37.751 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:50:37.767 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:50:37.787 INFO:tasks.ceph.osd.5:Stopped 2024-04-20T02:50:37.787 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 5 2024-04-20T02:50:37.788 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:50:37.808 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:50:37.842 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:37.842+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:37.860 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:37.860 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:37.871 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:50:37.943 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:37.944+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:38.043 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:38.043 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:38.044 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:38.045+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:38.053 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:50:38.142 INFO:tasks.ceph.mon.a.smithi105.stderr:2024-04-20T02:50:38.142+0000 7f7f0b14c700 -1 mon.a@0(leader).osd e25 definitely_dead 0 2024-04-20T02:50:38.144 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:38.145+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:38.182 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:38.182 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:38.192 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:50:38.245 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:38.246+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:38.259 INFO:teuthology.orchestra.run.smithi105.stderr:marked down osd.5. 2024-04-20T02:50:38.273 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-20T02:50:38.273 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-04-20T02:50:38.356 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:38.357+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:38.369 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:38.369 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:38.380 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:50:38.465 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:38.465+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:38.548 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:38.549+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:38.552 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:38.552 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:38.563 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:50:38.650 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:38.650+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:38.671 INFO:tasks.ceph.mon.a.smithi105.stderr:2024-04-20T02:50:38.671+0000 7f7f0b14c700 -1 mon.a@0(leader).osd e26 definitely_dead 0 2024-04-20T02:50:38.733 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:38.733 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:38.743 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:50:38.751 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:38.752+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:38.852 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:38.852+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:38.873 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:38.873 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:38.883 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:50:38.953 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:38.953+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:39.010 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:39.010 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:39.020 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:50:39.054 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:39.054+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:39.154 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:39.155+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:39.161 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:39.161 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:39.172 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:50:39.255 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:39.255+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:39.261 INFO:teuthology.orchestra.run.smithi105.stderr:marked out osd.5. 2024-04-20T02:50:39.269 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:50:39.273 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-04-20T02:50:39.273 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:50:39.273 DEBUG:teuthology.orchestra.run.smithi138:> type ceph-objectstore-tool 2024-04-20T02:50:39.290 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-20T02:50:39.291 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-04-20T02:50:39.347 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:39.348 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:39.358 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:39.358+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:39.359 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:50:39.459 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:39.460+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:39.529 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:39.529 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:39.539 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:50:39.561 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:39.560+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:39.661 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:39.661+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:39.714 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:39.714 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:39.730 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:50:39.762 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:39.763+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:39.860 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:39.860 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:39.860 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:39.860 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:39.861 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:39.863 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:39.863+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:39.870 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:50:39.964 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:39.964+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:40.000 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:35 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'21, mlcod=16'15)", 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.964670+0000", 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.035772671, 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054852659999999999, 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964670+0000", 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964670+0000", 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964670+0000", 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964691+0000", 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1175e-05 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964698+0000", 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3950000000000003e-06 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964704+0000", 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4829999999999999e-06 2024-04-20T02:50:40.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964747+0000", 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3016000000000003e-05 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.964780+0000", 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3000999999999998e-05 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.970129+0000", 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005349286 2024-04-20T02:50:40.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.970156+0000", 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.639e-05 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:106 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'68, mlcod=16'64)", 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.136358+0000", 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.864085111000001, 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049723069999999996, 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.005 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136358+0000", 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136358+0000", 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136357+0000", 2024-04-20T02:50:40.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136379+0000", 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1742999999999998e-05 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136388+0000", 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.268e-06 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136394+0000", 2024-04-20T02:50:40.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7089999999999999e-06 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136424+0000", 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0052e-05 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.136436+0000", 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1754e-05 2024-04-20T02:50:40.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141295+0000", 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004858996 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141330+0000", 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5383999999999999e-05 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.009 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:111 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'73, mlcod=16'65)", 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.138432+0000", 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.862010479999999, 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054900060000000004, 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138432+0000", 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138432+0000", 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138432+0000", 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138438+0000", 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1909999999999997e-06 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138448+0000", 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9660000000000008e-06 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138454+0000", 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4119999999999998e-06 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138504+0000", 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9301000000000001e-05 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138524+0000", 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0319e-05 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.143889+0000", 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053652079999999998 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.143922+0000", 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3229000000000003e-05 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.013 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:112 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'74, mlcod=16'65)", 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.138900+0000", 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.861542719999999, 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050438439999999996, 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138900+0000", 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138900+0000", 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138900+0000", 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138948+0000", 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8612999999999997e-05 2024-04-20T02:50:40.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138957+0000", 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5080000000000005e-06 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138962+0000", 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8099999999999997e-06 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.138994+0000", 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2697999999999998e-05 2024-04-20T02:50:40.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.139013+0000", 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8808e-05 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.143925+0000", 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049121440000000002 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.143944+0000", 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8853000000000001e-05 2024-04-20T02:50:40.017 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:121 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'83, mlcod=16'78)", 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.149068+0000", 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.851375156, 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050287819999999999, 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149068+0000", 2024-04-20T02:50:40.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149068+0000", 2024-04-20T02:50:40.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149067+0000", 2024-04-20T02:50:40.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149113+0000", 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6001999999999998e-05 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149122+0000", 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2329999999999995e-06 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149128+0000", 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.339e-06 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149170+0000", 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2088000000000001e-05 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.149182+0000", 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2653000000000001e-05 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.154064+0000", 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048819750000000002 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.154097+0000", 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2067000000000003e-05 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:131 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'89, mlcod=16'85)", 2024-04-20T02:50:40.022 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.476200+0000", 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.524242894, 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099670190000000006, 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476200+0000", 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476200+0000", 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476199+0000", 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476232+0000", 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2197999999999999e-05 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476239+0000", 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6389999999999992e-06 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476249+0000", 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7310000000000005e-06 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476280+0000", 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1417000000000001e-05 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476298+0000", 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7683999999999999e-05 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486147+0000", 2024-04-20T02:50:40.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0098485159999999999 2024-04-20T02:50:40.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486167+0000", 2024-04-20T02:50:40.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0373e-05 2024-04-20T02:50:40.026 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.026 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:132 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'90, mlcod=16'85)", 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.478573+0000", 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.521869663, 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0076055539999999996, 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478573+0000", 2024-04-20T02:50:40.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478573+0000", 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478573+0000", 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478577+0000", 2024-04-20T02:50:40.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7090000000000001e-06 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478585+0000", 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7100000000000007e-06 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478591+0000", 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2500000000000003e-06 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478656+0000", 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4111999999999999e-05 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478672+0000", 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6854000000000002e-05 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486169+0000", 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074966260000000002 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486179+0000", 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7520000000000008e-06 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:133 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'86)", 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.479193+0000", 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.521249921999999, 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069972599999999999, 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479193+0000", 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479193+0000", 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479192+0000", 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479197+0000", 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7829999999999998e-06 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479205+0000", 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7919999999999993e-06 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479210+0000", 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8570000000000003e-06 2024-04-20T02:50:40.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479238+0000", 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8501e-05 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479253+0000", 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4483e-05 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486180+0000", 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069272320000000002 2024-04-20T02:50:40.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486190+0000", 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0127e-05 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:134 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'86)", 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.479644+0000", 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.520798831, 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0065671790000000002, 2024-04-20T02:50:40.035 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479644+0000", 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479644+0000", 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479643+0000", 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479669+0000", 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5230000000000001e-05 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479678+0000", 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4930000000000004e-06 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479685+0000", 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5220000000000004e-06 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479727+0000", 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2583000000000002e-05 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479740+0000", 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.325e-05 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486192+0000", 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0064515759999999997 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486211+0000", 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9293e-05 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:135 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'86)", 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.480066+0000", 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.520376832, 2024-04-20T02:50:40.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.006170343, 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480066+0000", 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480066+0000", 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480066+0000", 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480070+0000", 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7110000000000003e-06 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480078+0000", 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7379999999999994e-06 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480083+0000", 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3210000000000004e-06 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480110+0000", 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6489999999999999e-05 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480131+0000", 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1039e-05 2024-04-20T02:50:40.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486216+0000", 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060846099999999998 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486236+0000", 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0832999999999999e-05 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:136 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'86)", 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.480471+0000", 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.519972106000001, 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057863059999999997, 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480471+0000", 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480471+0000", 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480470+0000", 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480478+0000", 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.011e-06 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480488+0000", 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3789999999999995e-06 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480494+0000", 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0850000000000002e-06 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480525+0000", 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1164999999999997e-05 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480540+0000", 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4946999999999999e-05 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486240+0000", 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0056999360000000001 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486257+0000", 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7326999999999999e-05 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.047 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:137 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'95, mlcod=16'86)", 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.480922+0000", 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.519521217000001, 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053589800000000002, 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480922+0000", 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480922+0000", 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480921+0000", 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480943+0000", 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1934999999999999e-05 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480951+0000", 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9699999999999999e-06 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480956+0000", 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9119999999999997e-06 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480993+0000", 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6956999999999999e-05 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481012+0000", 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8607e-05 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486259+0000", 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005247425 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486281+0000", 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1713e-05 2024-04-20T02:50:40.051 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:138 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'96, mlcod=16'86)", 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.481709+0000", 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.518734391999999, 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049116979999999999, 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481709+0000", 2024-04-20T02:50:40.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481709+0000", 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481708+0000", 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481712+0000", 2024-04-20T02:50:40.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3420000000000001e-06 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481719+0000", 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0469999999999996e-06 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481724+0000", 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2520000000000001e-06 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483637+0000", 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001912789 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483649+0000", 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2771e-05 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486578+0000", 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029286820000000002 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486620+0000", 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2224000000000001e-05 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:140 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'98, mlcod=16'87)", 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.482489+0000", 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.517953644999999, 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066787219999999998, 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.056 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482489+0000", 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482489+0000", 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482488+0000", 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:40.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482507+0000", 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9338999999999999e-05 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482521+0000", 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482527+0000", 2024-04-20T02:50:40.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7610000000000004e-06 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484064+0000", 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001536737 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484086+0000", 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2178999999999999e-05 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489141+0000", 2024-04-20T02:50:40.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050549540000000004 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489168+0000", 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7022e-05 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:141 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'87)", 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.483045+0000", 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.517397625000001, 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0061406789999999996, 2024-04-20T02:50:40.060 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483045+0000", 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483045+0000", 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483045+0000", 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483050+0000", 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3310000000000003e-06 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483059+0000", 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9230000000000008e-06 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483066+0000", 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3250000000000002e-06 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486305+0000", 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0032387090000000002 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486325+0000", 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9905e-05 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489170+0000", 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0028451230000000002 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489186+0000", 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5883e-05 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:155 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'109, mlcod=16'105)", 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.952241+0000", 2024-04-20T02:50:40.064 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.048201581000001, 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0056084400000000001, 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952241+0000", 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952241+0000", 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952241+0000", 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952298+0000", 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7089999999999997e-05 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952307+0000", 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7609999999999996e-06 2024-04-20T02:50:40.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952314+0000", 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9999999999999999e-06 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952348+0000", 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3797999999999997e-05 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952364+0000", 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6010000000000001e-05 2024-04-20T02:50:40.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957821+0000", 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005457249 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957850+0000", 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8969999999999999e-05 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.068 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:156 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'110, mlcod=16'105)", 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.952994+0000", 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.047449236999999, 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0070517490000000004, 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952994+0000", 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952994+0000", 2024-04-20T02:50:40.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952993+0000", 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953005+0000", 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1412e-05 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953013+0000", 2024-04-20T02:50:40.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0129999999999993e-06 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953019+0000", 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5760000000000002e-06 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.954938+0000", 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019186170000000001 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.954959+0000", 2024-04-20T02:50:40.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0721e-05 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960017+0000", 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00505843 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960045+0000", 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8450999999999999e-05 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:157 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'111, mlcod=16'105)", 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.953402+0000", 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.047040548999998, 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066620789999999996, 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953402+0000", 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953402+0000", 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953402+0000", 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953407+0000", 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1440000000000002e-06 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953413+0000", 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.981e-06 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953421+0000", 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6199999999999999e-06 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955173+0000", 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001752094 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955192+0000", 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9570000000000001e-05 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960047+0000", 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048545230000000003 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960064+0000", 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7533e-05 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=16'105)", 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.953833+0000", 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.046609790000002, 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0065076650000000002, 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953833+0000", 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953833+0000", 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953833+0000", 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953861+0000", 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8524999999999999e-05 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953867+0000", 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6500000000000001e-06 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953871+0000", 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0910000000000003e-06 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955472+0000", 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016007300000000001 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955495+0000", 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.317e-05 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960316+0000", 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048213479999999996 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960341+0000", 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4409999999999998e-05 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.080 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4201.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=16'105)", 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.955527+0000", 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.044915942999999, 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004827904, 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955527+0000", 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955527+0000", 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955526+0000", 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955534+0000", 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6680000000000001e-06 2024-04-20T02:50:40.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955545+0000", 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.149e-05 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955553+0000", 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6459999999999993e-06 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957592+0000", 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020391799999999998 2024-04-20T02:50:40.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957613+0000", 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1053e-05 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960343+0000", 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002729469 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960355+0000", 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2055000000000001e-05 2024-04-20T02:50:40.084 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.085 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.085 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.085 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.085 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.085 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:40.086 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:50:40.087 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:40.086+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:40.187 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:40.187+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:40.218 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:40.218 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:40.218 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:40.218 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:40.228 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:50:40.265 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:50:40.322 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:40.322+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:40.388 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:40.389+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:40.396 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:40.397 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:40.407 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:50:40.473 INFO:teuthology.orchestra.run.smithi138.stdout:4.8 2024-04-20T02:50:40.473 INFO:teuthology.orchestra.run.smithi138.stdout:4.7 2024-04-20T02:50:40.473 INFO:teuthology.orchestra.run.smithi138.stdout:4.3 2024-04-20T02:50:40.473 INFO:teuthology.orchestra.run.smithi138.stdout:4.2 2024-04-20T02:50:40.473 INFO:teuthology.orchestra.run.smithi138.stdout:4.1 2024-04-20T02:50:40.473 INFO:teuthology.orchestra.run.smithi138.stdout:3.2 2024-04-20T02:50:40.473 INFO:teuthology.orchestra.run.smithi138.stdout:2.3 2024-04-20T02:50:40.473 INFO:teuthology.orchestra.run.smithi138.stdout:2.2 2024-04-20T02:50:40.489 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:40.489+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:40.579 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:40.579 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:50:40.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:50:40.579 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:50:40.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.579 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:50:40.579 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:50:40.579 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.973621949999998, 2024-04-20T02:50:40.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:50:40.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:50:40.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.135845+0000", 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.441691708, 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0091040970000000002, 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.586 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 114 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135845+0000", 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135845+0000", 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135844+0000", 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135848+0000", 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2230000000000001e-06 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135854+0000", 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5679999999999999e-06 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135858+0000", 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1099999999999996e-06 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141165+0000", 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053073260000000002 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141221+0000", 2024-04-20T02:50:40.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5161000000000003e-05 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141303+0000", 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.2003999999999996e-05 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144686+0000", 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033838290000000001 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144857+0000", 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017099800000000001 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144906+0000", 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8812e-05 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144930+0000", 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4096000000000001e-05 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144949+0000", 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8479999999999999e-05 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.135864+0000", 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.441672946000001, 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0091993409999999998, 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.592 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 115 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135864+0000", 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135864+0000", 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135863+0000", 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135918+0000", 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.452e-05 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135925+0000", 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4410000000000004e-06 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135931+0000", 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8540000000000002e-06 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141573+0000", 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0056424960000000003 2024-04-20T02:50:40.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141629+0000", 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5260999999999998e-05 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.141742+0000", 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000112898 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144862+0000", 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003120795 2024-04-20T02:50:40.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144889+0000", 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6401e-05 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.144992+0000", 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103689 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145032+0000", 2024-04-20T02:50:40.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9167e-05 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145063+0000", 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1251999999999998e-05 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.135939+0000", 2024-04-20T02:50:40.598 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.441597703999999, 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010214235, 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 116 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135939+0000", 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135939+0000", 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135938+0000", 2024-04-20T02:50:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135943+0000", 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4159999999999997e-06 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135949+0000", 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2009999999999996e-06 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135954+0000", 2024-04-20T02:50:40.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2320000000000001e-06 2024-04-20T02:50:40.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145095+0000", 2024-04-20T02:50:40.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.009141415 2024-04-20T02:50:40.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145142+0000", 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6573000000000003e-05 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145224+0000", 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.2261000000000004e-05 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145863+0000", 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063818000000000002 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146001+0000", 2024-04-20T02:50:40.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013806300000000001 2024-04-20T02:50:40.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146084+0000", 2024-04-20T02:50:40.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3651000000000003e-05 2024-04-20T02:50:40.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146118+0000", 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3442000000000003e-05 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146153+0000", 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5345000000000001e-05 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.605 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.135960+0000", 2024-04-20T02:50:40.606 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.441576194, 2024-04-20T02:50:40.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010370449, 2024-04-20T02:50:40.606 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.606 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.606 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.606 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.606 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.606 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 117 2024-04-20T02:50:40.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.606 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135960+0000", 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135960+0000", 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135960+0000", 2024-04-20T02:50:40.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135966+0000", 2024-04-20T02:50:40.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9529999999999996e-06 2024-04-20T02:50:40.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135972+0000", 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.135978+0000", 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5060000000000003e-06 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145515+0000", 2024-04-20T02:50:40.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095370119999999992 2024-04-20T02:50:40.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145562+0000", 2024-04-20T02:50:40.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:50:40.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.145646+0000", 2024-04-20T02:50:40.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3327999999999999e-05 2024-04-20T02:50:40.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146255+0000", 2024-04-20T02:50:40.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00060893600000000003 2024-04-20T02:50:40.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146263+0000", 2024-04-20T02:50:40.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5429999999999994e-06 2024-04-20T02:50:40.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146285+0000", 2024-04-20T02:50:40.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1458999999999999e-05 2024-04-20T02:50:40.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146311+0000", 2024-04-20T02:50:40.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6526e-05 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.146331+0000", 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9678e-05 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:131 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475068+0000", 2024-04-20T02:50:40.613 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.102468962, 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011277362000000001, 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 131 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475068+0000", 2024-04-20T02:50:40.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475068+0000", 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475067+0000", 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475095+0000", 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8326e-05 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475104+0000", 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6729999999999998e-06 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475108+0000", 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.529e-06 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475874+0000", 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076629400000000005 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475955+0000", 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0963000000000004e-05 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.476098+0000", 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014343099999999999 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478942+0000", 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0028435259999999999 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481887+0000", 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029456679999999998 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486272+0000", 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004384125 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486308+0000", 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6786000000000003e-05 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486345+0000", 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6461000000000002e-05 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.619 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475113+0000", 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.102423006999999, 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011336222, 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 132 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475113+0000", 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475113+0000", 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475113+0000", 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475115+0000", 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2249999999999999e-06 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475119+0000", 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3289999999999998e-06 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475122+0000", 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9239999999999999e-06 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478386+0000", 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0032647829999999998 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478437+0000", 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0068000000000002e-05 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.478510+0000", 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3889999999999999e-05 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481487+0000", 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002976472 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481891+0000", 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040387700000000002 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486399+0000", 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004508359 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486428+0000", 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8852000000000001e-05 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486450+0000", 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1685999999999999e-05 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.625 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475313+0000", 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.102223419000001, 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011242378000000001, 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 133 2024-04-20T02:50:40.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475313+0000", 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475313+0000", 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475312+0000", 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475319+0000", 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8589999999999999e-06 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475330+0000", 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1209000000000001e-05 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475338+0000", 2024-04-20T02:50:40.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8560000000000007e-06 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479010+0000", 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003672219 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479057+0000", 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6684000000000002e-05 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479136+0000", 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8978000000000002e-05 2024-04-20T02:50:40.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481894+0000", 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0027578780000000001 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483367+0000", 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014729750000000001 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486484+0000", 2024-04-20T02:50:40.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0031168210000000001 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486522+0000", 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7936999999999999e-05 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486555+0000", 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3782000000000003e-05 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475361+0000", 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.102175619, 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011315524, 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 134 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475361+0000", 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475361+0000", 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475360+0000", 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475403+0000", 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2311000000000002e-05 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475410+0000", 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475416+0000", 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3219999999999998e-06 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479420+0000", 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0040038000000000001 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479471+0000", 2024-04-20T02:50:40.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1582999999999997e-05 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479548+0000", 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6747000000000002e-05 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481897+0000", 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0023488390000000001 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483386+0000", 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014890719999999999 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486610+0000", 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0032237810000000002 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486656+0000", 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5940999999999999e-05 2024-04-20T02:50:40.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486676+0000", 2024-04-20T02:50:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0608999999999999e-05 2024-04-20T02:50:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.638 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475426+0000", 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.102110233000001, 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011332076, 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 135 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475426+0000", 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475426+0000", 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475426+0000", 2024-04-20T02:50:40.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475430+0000", 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5639999999999998e-06 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475436+0000", 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7799999999999997e-06 2024-04-20T02:50:40.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475442+0000", 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479825+0000", 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0043834360000000001 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.479908+0000", 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3195000000000006e-05 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480007+0000", 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8997000000000001e-05 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481900+0000", 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0018928529999999999 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483406+0000", 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001505981 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486710+0000", 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033038949999999998 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486739+0000", 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9464000000000001e-05 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486758+0000", 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8969000000000001e-05 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.645 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475447+0000", 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.102089278000001, 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011384475, 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 136 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475447+0000", 2024-04-20T02:50:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475447+0000", 2024-04-20T02:50:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475447+0000", 2024-04-20T02:50:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475453+0000", 2024-04-20T02:50:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8409999999999996e-06 2024-04-20T02:50:40.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475461+0000", 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.2109999999999998e-06 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475466+0000", 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4809999999999997e-06 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480251+0000", 2024-04-20T02:50:40.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0047850510000000002 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480302+0000", 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0801999999999999e-05 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480377+0000", 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4772999999999997e-05 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481903+0000", 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001526052 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483768+0000", 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0018654089999999999 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486792+0000", 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003023141 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486814+0000", 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1942000000000001e-05 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486832+0000", 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8117e-05 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475472+0000", 2024-04-20T02:50:40.652 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.102064293000002, 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011424245, 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475472+0000", 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475472+0000", 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475472+0000", 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475539+0000", 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6761999999999995e-05 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475545+0000", 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5899999999999996e-06 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475551+0000", 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7760000000000003e-06 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480608+0000", 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050569439999999998 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480692+0000", 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3689000000000005e-05 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.480823+0000", 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013148399999999999 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483820+0000", 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029966509999999999 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484012+0000", 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019240199999999999 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486858+0000", 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002845991 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486880+0000", 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1503000000000001e-05 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486896+0000", 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6891000000000001e-05 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.658 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475557+0000", 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.101979322999998, 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011424037999999999, 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 138 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475557+0000", 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475557+0000", 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475557+0000", 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475561+0000", 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8890000000000004e-06 2024-04-20T02:50:40.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475574+0000", 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3006999999999999e-05 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475584+0000", 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.027e-05 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481511+0000", 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0059274560000000002 2024-04-20T02:50:40.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481576+0000", 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4918000000000004e-05 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481650+0000", 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3668e-05 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483879+0000", 2024-04-20T02:50:40.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00222923 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484022+0000", 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014319100000000001 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486943+0000", 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029209969999999998 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486964+0000", 2024-04-20T02:50:40.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0843999999999999e-05 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486981+0000", 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7076e-05 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475593+0000", 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.101943206000001, 2024-04-20T02:50:40.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011444096000000001, 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 139 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475593+0000", 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475593+0000", 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475593+0000", 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475597+0000", 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2080000000000002e-06 2024-04-20T02:50:40.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475602+0000", 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.271e-06 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475606+0000", 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.517e-06 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481928+0000", 2024-04-20T02:50:40.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.006322449 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.481976+0000", 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7840999999999998e-05 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482058+0000", 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.1842999999999996e-05 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483920+0000", 2024-04-20T02:50:40.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0018621149999999999 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484027+0000", 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106959 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.487000+0000", 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029727669999999999 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.487019+0000", 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9542999999999999e-05 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.487037+0000", 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8107000000000001e-05 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.670 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475613+0000", 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.101923062000001, 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013679606, 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 140 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475613+0000", 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475613+0000", 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475613+0000", 2024-04-20T02:50:40.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475665+0000", 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1641999999999998e-05 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475672+0000", 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5100000000000001e-06 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475677+0000", 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4290000000000003e-06 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482302+0000", 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066251310000000002 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482350+0000", 2024-04-20T02:50:40.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482422+0000", 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2260000000000003e-05 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.483956+0000", 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015343589999999999 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484032+0000", 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5877000000000006e-05 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489249+0000", 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0052166499999999998 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489275+0000", 2024-04-20T02:50:40.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6148000000000001e-05 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489293+0000", 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7961999999999998e-05 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.677 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.475681+0000", 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 20.101855523000001, 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013671255, 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 141 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475681+0000", 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475681+0000", 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475681+0000", 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475683+0000", 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4569999999999999e-06 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475686+0000", 2024-04-20T02:50:40.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0599999999999999e-06 2024-04-20T02:50:40.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.475689+0000", 2024-04-20T02:50:40.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.909e-06 2024-04-20T02:50:40.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482873+0000", 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0071836540000000003 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482921+0000", 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8637999999999997e-05 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.482995+0000", 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769000000000005e-05 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.484038+0000", 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010431349999999999 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.486921+0000", 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0028831360000000001 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489314+0000", 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002392936 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489336+0000", 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1648e-05 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.489352+0000", 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6240999999999999e-05 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:156 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-20T02:50:40.684 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.950943+0000", 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.626593695, 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092894120000000004, 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 156 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950943+0000", 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950943+0000", 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950942+0000", 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950953+0000", 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0438999999999999e-05 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950956+0000", 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.269e-06 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950960+0000", 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7290000000000002e-06 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952711+0000", 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017508910000000001 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952797+0000", 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6635e-05 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.952926+0000", 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000128746 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957448+0000", 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004521466 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.958448+0000", 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00099999799999999999 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960150+0000", 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001702582 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960196+0000", 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5766000000000002e-05 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960232+0000", 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6179e-05 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.950963+0000", 2024-04-20T02:50:40.690 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.626573566000001, 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0093661939999999996, 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 157 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950963+0000", 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950963+0000", 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950963+0000", 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950965+0000", 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9690000000000001e-06 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950967+0000", 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7049999999999999e-06 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950970+0000", 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7300000000000001e-06 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953211+0000", 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022404759999999999 2024-04-20T02:50:40.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953259+0000", 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8223000000000002e-05 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953338+0000", 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8949000000000006e-05 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.957455+0000", 2024-04-20T02:50:40.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041168819999999997 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.958468+0000", 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010132170000000001 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960266+0000", 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017985099999999999 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960295+0000", 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8753000000000001e-05 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960329+0000", 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4035000000000002e-05 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.950979+0000", 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.626557116000001, 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0094873709999999997, 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 158 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950979+0000", 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950979+0000", 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.950979+0000", 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951015+0000", 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6418999999999998e-05 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951020+0000", 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0600000000000001e-06 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951023+0000", 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7639999999999999e-06 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953612+0000", 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002588595 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953661+0000", 2024-04-20T02:50:40.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8893000000000001e-05 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.953739+0000", 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7818999999999995e-05 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.958484+0000", 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0047458129999999998 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.959996+0000", 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015118670000000001 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960409+0000", 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00041247799999999998 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960432+0000", 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3620000000000001e-05 2024-04-20T02:50:40.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960467+0000", 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4372999999999998e-05 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:20.951027+0000", 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.626509882000001, 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010295645000000001, 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:40.703 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 159 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951027+0000", 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951027+0000", 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951026+0000", 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:40.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951030+0000", 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2820000000000001e-06 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951032+0000", 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6979999999999998e-06 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.951035+0000", 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8100000000000002e-06 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955300+0000", 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0042653589999999998 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955348+0000", 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7414999999999999e-05 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.955427+0000", 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8737000000000002e-05 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960000+0000", 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004573293 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.960504+0000", 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050367299999999995 2024-04-20T02:50:40.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.961266+0000", 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076279099999999997 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.961288+0000", 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1814000000000002e-05 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:40.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:40.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:20.961322+0000", 2024-04-20T02:50:40.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4115000000000001e-05 2024-04-20T02:50:40.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.710 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:40.710 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:40.710 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:40.712 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:40.712+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:40.712 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:50:40.813 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:40.814+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:40.854 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:40.854 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:40.865 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:50:40.914 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:40.915+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:40.992 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:40.992 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:40.995 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 4.2 --file /var/log/ceph/exp.4.2.5 2024-04-20T02:50:41.002 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:50:41.047 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:41.047+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:41.116 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:41.116+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:41.132 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:41.132 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:41.142 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:50:41.226 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:41.226+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:41.261 INFO:tasks.rados.rados.0.smithi105.stdout:1: write initial oid 1 2024-04-20T02:50:41.261 INFO:tasks.rados.rados.0.smithi105.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-04-20T02:50:41.265 INFO:tasks.rados.rados.0.smithi105.stdout:1: writing smithi10548823-1 from 416807 to 892056 tid 1 2024-04-20T02:50:41.271 INFO:tasks.rados.rados.0.smithi105.stdout:1: writing smithi10548823-1 from 1542129 to 2016807 tid 2 2024-04-20T02:50:41.272 INFO:tasks.rados.rados.0.smithi105.stdout:2: write initial oid 2 2024-04-20T02:50:41.272 INFO:tasks.rados.rados.0.smithi105.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-04-20T02:50:41.277 INFO:tasks.rados.rados.0.smithi105.stdout:2: writing smithi10548823-2 from 416807 to 892056 tid 1 2024-04-20T02:50:41.283 INFO:tasks.rados.rados.0.smithi105.stdout:2: writing smithi10548823-2 from 1542129 to 2016807 tid 2 2024-04-20T02:50:41.283 INFO:tasks.rados.rados.0.smithi105.stdout:3: write initial oid 3 2024-04-20T02:50:41.284 INFO:tasks.rados.rados.0.smithi105.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-04-20T02:50:41.290 INFO:tasks.rados.rados.0.smithi105.stdout:3: writing smithi10548823-3 from 433614 to 984112 tid 1 2024-04-20T02:50:41.296 INFO:tasks.rados.rados.0.smithi105.stdout:3: writing smithi10548823-3 from 1600611 to 2033614 tid 2 2024-04-20T02:50:41.296 INFO:tasks.rados.rados.0.smithi105.stdout:4: write initial oid 4 2024-04-20T02:50:41.296 INFO:tasks.rados.rados.0.smithi105.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-04-20T02:50:41.304 INFO:tasks.rados.rados.0.smithi105.stdout:4: writing smithi10548823-4 from 450421 to 1076168 tid 1 2024-04-20T02:50:41.309 INFO:tasks.rados.rados.0.smithi105.stdout:4: writing smithi10548823-4 from 1659093 to 2050421 tid 2 2024-04-20T02:50:41.309 INFO:tasks.rados.rados.0.smithi105.stdout:5: write initial oid 5 2024-04-20T02:50:41.309 INFO:tasks.rados.rados.0.smithi105.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-04-20T02:50:41.316 INFO:tasks.rados.rados.0.smithi105.stdout:5: writing smithi10548823-5 from 467228 to 1168224 tid 1 2024-04-20T02:50:41.318 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:41.318+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:41.327 INFO:tasks.rados.rados.0.smithi105.stdout:5: writing smithi10548823-5 from 1717575 to 2067228 tid 2 2024-04-20T02:50:41.327 INFO:tasks.rados.rados.0.smithi105.stdout:6: write initial oid 6 2024-04-20T02:50:41.327 INFO:tasks.rados.rados.0.smithi105.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-04-20T02:50:41.331 INFO:tasks.rados.rados.0.smithi105.stdout:6: writing smithi10548823-6 from 484035 to 1260280 tid 1 2024-04-20T02:50:41.332 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:41.332 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:41.335 INFO:tasks.rados.rados.0.smithi105.stdout:6: writing smithi10548823-6 from 2059704 to 2084035 tid 2 2024-04-20T02:50:41.336 INFO:tasks.rados.rados.0.smithi105.stdout:7: write initial oid 7 2024-04-20T02:50:41.336 INFO:tasks.rados.rados.0.smithi105.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-04-20T02:50:41.341 INFO:tasks.rados.rados.0.smithi105.stdout:7: writing smithi10548823-7 from 500842 to 952336 tid 1 2024-04-20T02:50:41.349 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:50:41.349 INFO:tasks.rados.rados.0.smithi105.stdout:7: writing smithi10548823-7 from 1718186 to 2100842 tid 2 2024-04-20T02:50:41.349 INFO:tasks.rados.rados.0.smithi105.stdout:8: write initial oid 8 2024-04-20T02:50:41.349 INFO:tasks.rados.rados.0.smithi105.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-04-20T02:50:41.354 INFO:tasks.rados.rados.0.smithi105.stdout:8: writing smithi10548823-8 from 517649 to 1044392 tid 1 2024-04-20T02:50:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:8: writing smithi10548823-8 from 1776668 to 2117649 tid 2 2024-04-20T02:50:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:9: write initial oid 9 2024-04-20T02:50:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-04-20T02:50:41.369 INFO:tasks.rados.rados.0.smithi105.stdout:9: writing smithi10548823-9 from 534456 to 1252801 tid 1 2024-04-20T02:50:41.374 INFO:tasks.rados.rados.0.smithi105.stdout:9: writing smithi10548823-9 from 1951503 to 2134456 tid 2 2024-04-20T02:50:41.375 INFO:tasks.rados.rados.0.smithi105.stdout:10: write initial oid 10 2024-04-20T02:50:41.375 INFO:tasks.rados.rados.0.smithi105.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-04-20T02:50:41.382 INFO:tasks.rados.rados.0.smithi105.stdout:10: writing smithi10548823-10 from 551263 to 1344857 tid 1 2024-04-20T02:50:41.387 INFO:tasks.rados.rados.0.smithi105.stdout:10: writing smithi10548823-10 from 1893632 to 2151263 tid 2 2024-04-20T02:50:41.394 INFO:tasks.rados.rados.0.smithi105.stdout:11: write initial oid 11 2024-04-20T02:50:41.394 INFO:tasks.rados.rados.0.smithi105.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-04-20T02:50:41.395 INFO:tasks.rados.rados.0.smithi105.stdout:11: writing smithi10548823-11 from 568070 to 1036913 tid 1 2024-04-20T02:50:41.402 INFO:tasks.rados.rados.0.smithi105.stdout:11: writing smithi10548823-11 from 1552114 to 2168070 tid 2 2024-04-20T02:50:41.412 INFO:tasks.rados.rados.0.smithi105.stdout:12: write initial oid 12 2024-04-20T02:50:41.412 INFO:tasks.rados.rados.0.smithi105.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-04-20T02:50:41.413 INFO:tasks.rados.rados.0.smithi105.stdout:12: writing smithi10548823-12 from 584877 to 1128969 tid 1 2024-04-20T02:50:41.419 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:41.419+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:41.426 INFO:tasks.rados.rados.0.smithi105.stdout:12: writing smithi10548823-12 from 1610596 to 2172599 tid 2 2024-04-20T02:50:41.426 INFO:tasks.rados.rados.0.smithi105.stdout:13: write initial oid 13 2024-04-20T02:50:41.426 INFO:tasks.rados.rados.0.smithi105.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-04-20T02:50:41.427 INFO:tasks.rados.rados.0.smithi105.stdout:13: writing smithi10548823-13 from 601684 to 1221025 tid 1 2024-04-20T02:50:41.437 INFO:tasks.rados.rados.0.smithi105.stdout:13: writing smithi10548823-13 from 1669078 to 2201684 tid 2 2024-04-20T02:50:41.438 INFO:tasks.rados.rados.0.smithi105.stdout:14: write initial oid 14 2024-04-20T02:50:41.438 INFO:tasks.rados.rados.0.smithi105.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-04-20T02:50:41.439 INFO:tasks.rados.rados.0.smithi105.stdout:14: writing smithi10548823-14 from 618491 to 1313081 tid 1 2024-04-20T02:50:41.444 INFO:tasks.rados.rados.0.smithi105.stdout:14: writing smithi10548823-14 from 2011207 to 2218491 tid 2 2024-04-20T02:50:41.444 INFO:tasks.rados.rados.0.smithi105.stdout:15: write initial oid 15 2024-04-20T02:50:41.444 INFO:tasks.rados.rados.0.smithi105.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-04-20T02:50:41.452 INFO:tasks.rados.rados.0.smithi105.stdout:15: writing smithi10548823-15 from 635298 to 1405137 tid 1 2024-04-20T02:50:41.457 INFO:tasks.rados.rados.0.smithi105.stdout:15: writing smithi10548823-15 from 2069689 to 2235298 tid 2 2024-04-20T02:50:41.457 INFO:tasks.rados.rados.0.smithi105.stdout:16: write initial oid 16 2024-04-20T02:50:41.457 INFO:tasks.rados.rados.0.smithi105.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-04-20T02:50:41.469 INFO:tasks.rados.rados.0.smithi105.stdout:16: writing smithi10548823-16 from 652105 to 1097193 tid 1 2024-04-20T02:50:41.470 INFO:tasks.rados.rados.0.smithi105.stdout:16: writing smithi10548823-16 from 1728171 to 2181159 tid 2 2024-04-20T02:50:41.470 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:1: finishing write tid 1 to smithi10548823-1 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:1: finishing write tid 2 to smithi10548823-1 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:1: finishing write tid 3 to smithi10548823-1 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:1: finishing write tid 4 to smithi10548823-1 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:1: oid 1 updating version 0 to 2 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:1: oid 1 version 2 is already newer than 1 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:1: oid 1 updating version 2 to 3 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:2: finishing write tid 1 to smithi10548823-2 2024-04-20T02:50:41.471 INFO:tasks.rados.rados.0.smithi105.stdout:2: finishing write tid 2 to smithi10548823-2 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:3: finishing write tid 1 to smithi10548823-3 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:2: finishing write tid 3 to smithi10548823-2 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:2: finishing write tid 4 to smithi10548823-2 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:2: oid 2 updating version 0 to 1 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:2: oid 2 updating version 1 to 2 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:2: oid 2 updating version 2 to 3 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:3: finishing write tid 2 to smithi10548823-3 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:3: finishing write tid 3 to smithi10548823-3 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:3: finishing write tid 4 to smithi10548823-3 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:3: oid 3 updating version 0 to 1 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:3: oid 3 updating version 1 to 2 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:3: oid 3 updating version 2 to 3 2024-04-20T02:50:41.472 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:5: finishing write tid 1 to smithi10548823-5 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:5: finishing write tid 2 to smithi10548823-5 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:5: finishing write tid 3 to smithi10548823-5 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:5: finishing write tid 4 to smithi10548823-5 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:5: oid 5 updating version 0 to 2 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:5: oid 5 updating version 2 to 3 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:5: oid 5 version 3 is already newer than 1 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:4: finishing write tid 1 to smithi10548823-4 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:8: finishing write tid 1 to smithi10548823-8 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:8: finishing write tid 2 to smithi10548823-8 2024-04-20T02:50:41.473 INFO:tasks.rados.rados.0.smithi105.stdout:8: finishing write tid 3 to smithi10548823-8 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:8: finishing write tid 4 to smithi10548823-8 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:8: oid 8 updating version 0 to 2 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:8: oid 8 updating version 2 to 3 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:8: oid 8 version 3 is already newer than 1 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:4: finishing write tid 2 to smithi10548823-4 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:4: finishing write tid 3 to smithi10548823-4 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:6: finishing write tid 1 to smithi10548823-6 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:6: finishing write tid 2 to smithi10548823-6 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:6: finishing write tid 3 to smithi10548823-6 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:6: finishing write tid 4 to smithi10548823-6 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:6: oid 6 updating version 0 to 2 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:6: oid 6 updating version 2 to 3 2024-04-20T02:50:41.474 INFO:tasks.rados.rados.0.smithi105.stdout:6: oid 6 version 3 is already newer than 1 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:4: finishing write tid 4 to smithi10548823-4 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:4: oid 4 updating version 0 to 6 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:4: oid 4 version 6 is already newer than 5 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:4: oid 4 version 6 is already newer than 4 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:7: finishing write tid 1 to smithi10548823-7 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:7: finishing write tid 2 to smithi10548823-7 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:7: finishing write tid 3 to smithi10548823-7 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:7: finishing write tid 4 to smithi10548823-7 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:7: oid 7 updating version 0 to 7 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:7: oid 7 updating version 7 to 9 2024-04-20T02:50:41.475 INFO:tasks.rados.rados.0.smithi105.stdout:7: oid 7 version 9 is already newer than 8 2024-04-20T02:50:41.476 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 9 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:41.476 INFO:tasks.rados.rados.0.smithi105.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-20T02:50:41.476 INFO:tasks.rados.rados.0.smithi105.stdout:10: finishing write tid 1 to smithi10548823-10 2024-04-20T02:50:41.476 INFO:tasks.rados.rados.0.smithi105.stdout:10: finishing write tid 2 to smithi10548823-10 2024-04-20T02:50:41.476 INFO:tasks.rados.rados.0.smithi105.stdout:10: finishing write tid 3 to smithi10548823-10 2024-04-20T02:50:41.476 INFO:tasks.rados.rados.0.smithi105.stdout:10: finishing write tid 4 to smithi10548823-10 2024-04-20T02:50:41.476 INFO:tasks.rados.rados.0.smithi105.stdout:10: oid 10 updating version 0 to 5 2024-04-20T02:50:41.476 INFO:tasks.rados.rados.0.smithi105.stdout:10: oid 10 updating version 5 to 6 2024-04-20T02:50:41.476 INFO:tasks.rados.rados.0.smithi105.stdout:10: oid 10 version 6 is already newer than 4 2024-04-20T02:50:41.476 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:9: finishing write tid 1 to smithi10548823-9 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:9: finishing write tid 2 to smithi10548823-9 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:9: finishing write tid 3 to smithi10548823-9 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:9: finishing write tid 4 to smithi10548823-9 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:9: oid 9 updating version 0 to 10 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:9: oid 9 updating version 10 to 11 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:9: oid 9 updating version 11 to 12 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 12 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-20T02:50:41.477 INFO:tasks.rados.rados.0.smithi105.stdout:11: finishing write tid 1 to smithi10548823-11 2024-04-20T02:50:41.481 INFO:tasks.rados.rados.0.smithi105.stdout:1: done (15 left) 2024-04-20T02:50:41.481 INFO:tasks.rados.rados.0.smithi105.stdout:2: done (14 left) 2024-04-20T02:50:41.481 INFO:tasks.rados.rados.0.smithi105.stdout:3: done (13 left) 2024-04-20T02:50:41.481 INFO:tasks.rados.rados.0.smithi105.stdout:4: done (12 left) 2024-04-20T02:50:41.482 INFO:tasks.rados.rados.0.smithi105.stdout:5: done (11 left) 2024-04-20T02:50:41.482 INFO:tasks.rados.rados.0.smithi105.stdout:6: done (10 left) 2024-04-20T02:50:41.482 INFO:tasks.rados.rados.0.smithi105.stdout:7: done (9 left) 2024-04-20T02:50:41.482 INFO:tasks.rados.rados.0.smithi105.stdout:8: done (8 left) 2024-04-20T02:50:41.482 INFO:tasks.rados.rados.0.smithi105.stdout:9: done (7 left) 2024-04-20T02:50:41.482 INFO:tasks.rados.rados.0.smithi105.stdout:10: done (6 left) 2024-04-20T02:50:41.482 INFO:tasks.rados.rados.0.smithi105.stdout:17: write initial oid 17 2024-04-20T02:50:41.482 INFO:tasks.rados.rados.0.smithi105.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-04-20T02:50:41.482 INFO:tasks.rados.rados.0.smithi105.stdout:17: writing smithi10548823-17 from 668912 to 1305602 tid 1 2024-04-20T02:50:41.485 INFO:tasks.rados.rados.0.smithi105.stdout:17: writing smithi10548823-17 from 1903006 to 2268912 tid 2 2024-04-20T02:50:41.485 INFO:tasks.rados.rados.0.smithi105.stdout:18: write initial oid 18 2024-04-20T02:50:41.485 INFO:tasks.rados.rados.0.smithi105.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-04-20T02:50:41.492 INFO:tasks.rados.rados.0.smithi105.stdout:18: writing smithi10548823-18 from 685719 to 1397658 tid 1 2024-04-20T02:50:41.498 INFO:tasks.rados.rados.0.smithi105.stdout:18: writing smithi10548823-18 from 1845135 to 2285719 tid 2 2024-04-20T02:50:41.498 INFO:tasks.rados.rados.0.smithi105.stdout:19: write initial oid 19 2024-04-20T02:50:41.498 INFO:tasks.rados.rados.0.smithi105.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-04-20T02:50:41.506 INFO:tasks.rados.rados.0.smithi105.stdout:19: writing smithi10548823-19 from 702526 to 1489714 tid 1 2024-04-20T02:50:41.511 INFO:tasks.rados.rados.0.smithi105.stdout:19: writing smithi10548823-19 from 1903617 to 2302526 tid 2 2024-04-20T02:50:41.511 INFO:tasks.rados.rados.0.smithi105.stdout:20: write initial oid 20 2024-04-20T02:50:41.512 INFO:tasks.rados.rados.0.smithi105.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-04-20T02:50:41.517 INFO:tasks.rados.rados.0.smithi105.stdout:20: writing smithi10548823-20 from 719333 to 1181770 tid 1 2024-04-20T02:50:41.524 INFO:tasks.rados.rados.0.smithi105.stdout:20: writing smithi10548823-20 from 1962099 to 2319333 tid 2 2024-04-20T02:50:41.525 INFO:tasks.rados.rados.0.smithi105.stdout:21: write initial oid 21 2024-04-20T02:50:41.525 INFO:tasks.rados.rados.0.smithi105.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-04-20T02:50:41.531 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:41.520+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:41.532 INFO:tasks.rados.rados.0.smithi105.stdout:21: writing smithi10548823-21 from 736140 to 1273826 tid 1 2024-04-20T02:50:41.540 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:41.540 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:41.541 INFO:tasks.rados.rados.0.smithi105.stdout:21: writing smithi10548823-21 from 2020581 to 2336140 tid 2 2024-04-20T02:50:41.541 INFO:tasks.rados.rados.0.smithi105.stdout:22: write initial oid 22 2024-04-20T02:50:41.541 INFO:tasks.rados.rados.0.smithi105.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-04-20T02:50:41.547 INFO:tasks.rados.rados.0.smithi105.stdout:22: writing smithi10548823-22 from 752947 to 1365882 tid 1 2024-04-20T02:50:41.548 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:50:41.554 INFO:tasks.rados.rados.0.smithi105.stdout:22: writing smithi10548823-22 from 1962710 to 2352947 tid 2 2024-04-20T02:50:41.563 INFO:tasks.rados.rados.0.smithi105.stdout:23: write initial oid 23 2024-04-20T02:50:41.563 INFO:tasks.rados.rados.0.smithi105.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-04-20T02:50:41.566 INFO:tasks.rados.rados.0.smithi105.stdout:23: writing smithi10548823-23 from 769754 to 1457938 tid 1 2024-04-20T02:50:41.569 INFO:tasks.rados.rados.0.smithi105.stdout:23: writing smithi10548823-23 from 2021192 to 2369754 tid 2 2024-04-20T02:50:41.572 INFO:tasks.rados.rados.0.smithi105.stdout:24: write initial oid 24 2024-04-20T02:50:41.572 INFO:tasks.rados.rados.0.smithi105.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-04-20T02:50:41.576 INFO:tasks.rados.rados.0.smithi105.stdout:24: writing smithi10548823-24 from 786561 to 1266347 tid 1 2024-04-20T02:50:41.582 INFO:tasks.rados.rados.0.smithi105.stdout:24: writing smithi10548823-24 from 1796027 to 2263691 tid 2 2024-04-20T02:50:41.583 INFO:tasks.rados.rados.0.smithi105.stdout:25: write initial oid 25 2024-04-20T02:50:41.583 INFO:tasks.rados.rados.0.smithi105.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-04-20T02:50:41.589 INFO:tasks.rados.rados.0.smithi105.stdout:25: writing smithi10548823-25 from 403368 to 958403 tid 1 2024-04-20T02:50:41.597 INFO:tasks.rados.rados.0.smithi105.stdout:25: writing smithi10548823-25 from 1454509 to 2182184 tid 2 2024-04-20T02:50:41.597 INFO:tasks.rados.rados.0.smithi105.stdout:26: write initial oid 26 2024-04-20T02:50:41.597 INFO:tasks.rados.rados.0.smithi105.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-04-20T02:50:41.608 INFO:tasks.rados.rados.0.smithi105.stdout:26: writing smithi10548823-26 from 420175 to 1050459 tid 1 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:26: writing smithi10548823-26 from 1796638 to 2267971 tid 2 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:11: finishing write tid 2 to smithi10548823-11 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:11: finishing write tid 3 to smithi10548823-11 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:11: finishing write tid 4 to smithi10548823-11 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:11: oid 11 updating version 0 to 2 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:11: oid 11 version 2 is already newer than 1 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:11: oid 11 updating version 2 to 3 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:14: finishing write tid 1 to smithi10548823-14 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:14: finishing write tid 2 to smithi10548823-14 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:12: finishing write tid 1 to smithi10548823-12 2024-04-20T02:50:41.615 INFO:tasks.rados.rados.0.smithi105.stdout:12: finishing write tid 2 to smithi10548823-12 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:12: finishing write tid 3 to smithi10548823-12 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:12: finishing write tid 4 to smithi10548823-12 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:12: oid 12 updating version 0 to 1 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:12: oid 12 updating version 1 to 3 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:12: oid 12 version 3 is already newer than 2 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:14: finishing write tid 3 to smithi10548823-14 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:14: finishing write tid 4 to smithi10548823-14 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:14: oid 14 updating version 0 to 6 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:14: oid 14 version 6 is already newer than 5 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:14: oid 14 version 6 is already newer than 4 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-20T02:50:41.616 INFO:tasks.rados.rados.0.smithi105.stdout:17: finishing write tid 1 to smithi10548823-17 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:13: finishing write tid 1 to smithi10548823-13 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:13: finishing write tid 2 to smithi10548823-13 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:13: finishing write tid 3 to smithi10548823-13 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:13: finishing write tid 4 to smithi10548823-13 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:13: oid 13 updating version 0 to 6 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:13: oid 13 version 6 is already newer than 5 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:13: oid 13 version 6 is already newer than 4 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:15: finishing write tid 1 to smithi10548823-15 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:17: finishing write tid 2 to smithi10548823-17 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:17: finishing write tid 3 to smithi10548823-17 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:15: finishing write tid 2 to smithi10548823-15 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:15: finishing write tid 3 to smithi10548823-15 2024-04-20T02:50:41.617 INFO:tasks.rados.rados.0.smithi105.stdout:17: finishing write tid 4 to smithi10548823-17 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:17: oid 17 updating version 0 to 5 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:17: oid 17 version 5 is already newer than 4 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:17: oid 17 updating version 5 to 6 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:18: finishing write tid 1 to smithi10548823-18 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:18: finishing write tid 2 to smithi10548823-18 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:16: finishing write tid 1 to smithi10548823-16 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:16: finishing write tid 2 to smithi10548823-16 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:15: finishing write tid 4 to smithi10548823-15 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:15: oid 15 updating version 0 to 9 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:15: oid 15 version 9 is already newer than 8 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:15: oid 15 version 9 is already newer than 7 2024-04-20T02:50:41.618 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:18: finishing write tid 3 to smithi10548823-18 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:18: finishing write tid 4 to smithi10548823-18 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:18: oid 18 updating version 0 to 2 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:18: oid 18 version 2 is already newer than 1 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:18: oid 18 updating version 2 to 3 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:20: finishing write tid 1 to smithi10548823-20 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:16: finishing write tid 3 to smithi10548823-16 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:16: finishing write tid 4 to smithi10548823-16 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:16: oid 16 updating version 0 to 8 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:16: oid 16 version 8 is already newer than 7 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:16: oid 16 updating version 8 to 9 2024-04-20T02:50:41.619 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 9 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:19: finishing write tid 1 to smithi10548823-19 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:20: finishing write tid 2 to smithi10548823-20 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:20: finishing write tid 3 to smithi10548823-20 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:20: finishing write tid 4 to smithi10548823-20 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:20: oid 20 updating version 0 to 6 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:20: oid 20 version 6 is already newer than 4 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:20: oid 20 version 6 is already newer than 5 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:19: finishing write tid 2 to smithi10548823-19 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:19: finishing write tid 3 to smithi10548823-19 2024-04-20T02:50:41.620 INFO:tasks.rados.rados.0.smithi105.stdout:19: finishing write tid 4 to smithi10548823-19 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:19: oid 19 updating version 0 to 11 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:19: oid 19 version 11 is already newer than 10 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:19: oid 19 updating version 11 to 12 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 12 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:21: finishing write tid 1 to smithi10548823-21 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:21: finishing write tid 2 to smithi10548823-21 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:21: finishing write tid 3 to smithi10548823-21 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:21: finishing write tid 4 to smithi10548823-21 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:21: oid 21 updating version 0 to 8 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:21: oid 21 version 8 is already newer than 7 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:21: oid 21 updating version 8 to 9 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:41.621 INFO:tasks.rados.rados.0.smithi105.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:11: done (15 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:12: done (14 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:13: done (13 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:14: done (12 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:15: done (11 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:16: done (10 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:17: done (9 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:18: done (8 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:19: done (7 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:20: done (6 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:21: done (5 left) 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:27: write initial oid 27 2024-04-20T02:50:41.622 INFO:tasks.rados.rados.0.smithi105.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-04-20T02:50:41.623 INFO:tasks.rados.rados.0.smithi105.stdout:27: writing smithi10548823-27 from 436982 to 1142515 tid 1 2024-04-20T02:50:41.623 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:41.624+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:41.628 INFO:tasks.rados.rados.0.smithi105.stdout:27: writing smithi10548823-27 from 1855120 to 2436982 tid 2 2024-04-20T02:50:41.628 INFO:tasks.rados.rados.0.smithi105.stdout:28: write initial oid 28 2024-04-20T02:50:41.628 INFO:tasks.rados.rados.0.smithi105.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-04-20T02:50:41.636 INFO:tasks.rados.rados.0.smithi105.stdout:28: writing smithi10548823-28 from 453789 to 1234571 tid 1 2024-04-20T02:50:41.643 INFO:tasks.rados.rados.0.smithi105.stdout:28: writing smithi10548823-28 from 1913602 to 2388604 tid 2 2024-04-20T02:50:41.643 INFO:tasks.rados.rados.0.smithi105.stdout:29: write initial oid 29 2024-04-20T02:50:41.643 INFO:tasks.rados.rados.0.smithi105.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-04-20T02:50:41.653 INFO:tasks.rados.rados.0.smithi105.stdout:29: writing smithi10548823-29 from 470596 to 926627 tid 1 2024-04-20T02:50:41.663 INFO:tasks.rados.rados.0.smithi105.stdout:29: writing smithi10548823-29 from 1572084 to 2190744 tid 2 2024-04-20T02:50:41.663 INFO:tasks.rados.rados.0.smithi105.stdout:30: write initial oid 30 2024-04-20T02:50:41.663 INFO:tasks.rados.rados.0.smithi105.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-04-20T02:50:41.664 INFO:tasks.rados.rados.0.smithi105.stdout:30: writing smithi10548823-30 from 487403 to 1018683 tid 1 2024-04-20T02:50:41.674 INFO:tasks.rados.rados.0.smithi105.stdout:30: writing smithi10548823-30 from 1514213 to 1992884 tid 2 2024-04-20T02:50:41.674 INFO:tasks.rados.rados.0.smithi105.stdout:31: write initial oid 31 2024-04-20T02:50:41.675 INFO:tasks.rados.rados.0.smithi105.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-04-20T02:50:41.677 INFO:tasks.rados.rados.0.smithi105.stdout:31: writing smithi10548823-31 from 504210 to 1110739 tid 1 2024-04-20T02:50:41.685 INFO:tasks.rados.rados.0.smithi105.stdout:31: writing smithi10548823-31 from 1572695 to 2195024 tid 2 2024-04-20T02:50:41.685 INFO:tasks.rados.rados.0.smithi105.stdout:32: write initial oid 32 2024-04-20T02:50:41.686 INFO:tasks.rados.rados.0.smithi105.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-04-20T02:50:41.691 INFO:tasks.rados.rados.0.smithi105.stdout:32: writing smithi10548823-32 from 521017 to 1319148 tid 1 2024-04-20T02:50:41.698 INFO:tasks.rados.rados.0.smithi105.stdout:32: writing smithi10548823-32 from 1747530 to 2229870 tid 2 2024-04-20T02:50:41.698 INFO:tasks.rados.rados.0.smithi105.stdout:33: write initial oid 33 2024-04-20T02:50:41.698 INFO:tasks.rados.rados.0.smithi105.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-04-20T02:50:41.704 INFO:tasks.rados.rados.0.smithi105.stdout:33: writing smithi10548823-33 from 537824 to 1011204 tid 1 2024-04-20T02:50:41.712 INFO:tasks.rados.rados.0.smithi105.stdout:33: writing smithi10548823-33 from 1806012 to 2432010 tid 2 2024-04-20T02:50:41.712 INFO:tasks.rados.rados.0.smithi105.stdout:34: write initial oid 34 2024-04-20T02:50:41.712 INFO:tasks.rados.rados.0.smithi105.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-04-20T02:50:41.719 INFO:tasks.rados.rados.0.smithi105.stdout:34: writing smithi10548823-34 from 554631 to 1103260 tid 1 2024-04-20T02:50:41.725 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:41.725+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:41.727 INFO:tasks.rados.rados.0.smithi105.stdout:34: writing smithi10548823-34 from 1748141 to 2234150 tid 2 2024-04-20T02:50:41.727 INFO:tasks.rados.rados.0.smithi105.stdout:35: write initial oid 35 2024-04-20T02:50:41.727 INFO:tasks.rados.rados.0.smithi105.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-04-20T02:50:41.733 INFO:tasks.rados.rados.0.smithi105.stdout:35: writing smithi10548823-35 from 571438 to 1195316 tid 1 2024-04-20T02:50:41.740 INFO:tasks.rados.rados.0.smithi105.stdout:35: writing smithi10548823-35 from 1806623 to 2436290 tid 2 2024-04-20T02:50:41.740 INFO:tasks.rados.rados.0.smithi105.stdout:36: write initial oid 36 2024-04-20T02:50:41.740 INFO:tasks.rados.rados.0.smithi105.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-04-20T02:50:41.747 INFO:tasks.rados.rados.0.smithi105.stdout:36: writing smithi10548823-36 from 588245 to 1287372 tid 1 2024-04-20T02:50:41.753 INFO:tasks.rados.rados.0.smithi105.stdout:36: writing smithi10548823-36 from 1865105 to 2354783 tid 2 2024-04-20T02:50:41.753 INFO:tasks.rados.rados.0.smithi105.stdout:37: write initial oid 37 2024-04-20T02:50:41.753 INFO:tasks.rados.rados.0.smithi105.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-04-20T02:50:41.762 INFO:tasks.rados.rados.0.smithi105.stdout:37: writing smithi10548823-37 from 605052 to 1379428 tid 1 2024-04-20T02:50:41.772 INFO:tasks.rados.rados.0.smithi105.stdout:37: writing smithi10548823-37 from 1923587 to 2556923 tid 2 2024-04-20T02:50:41.772 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:41.772 INFO:tasks.rados.rados.0.smithi105.stdout:22: finishing write tid 1 to smithi10548823-22 2024-04-20T02:50:41.772 INFO:tasks.rados.rados.0.smithi105.stdout:22: finishing write tid 2 to smithi10548823-22 2024-04-20T02:50:41.772 INFO:tasks.rados.rados.0.smithi105.stdout:22: finishing write tid 3 to smithi10548823-22 2024-04-20T02:50:41.772 INFO:tasks.rados.rados.0.smithi105.stdout:22: finishing write tid 4 to smithi10548823-22 2024-04-20T02:50:41.772 INFO:tasks.rados.rados.0.smithi105.stdout:22: oid 22 updating version 0 to 3 2024-04-20T02:50:41.772 INFO:tasks.rados.rados.0.smithi105.stdout:22: oid 22 version 3 is already newer than 1 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:22: oid 22 version 3 is already newer than 2 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:25: finishing write tid 1 to smithi10548823-25 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:23: finishing write tid 1 to smithi10548823-23 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:23: finishing write tid 2 to smithi10548823-23 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:23: finishing write tid 3 to smithi10548823-23 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:23: finishing write tid 4 to smithi10548823-23 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:23: oid 23 updating version 0 to 6 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:23: oid 23 version 6 is already newer than 4 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:23: oid 23 version 6 is already newer than 5 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-20T02:50:41.773 INFO:tasks.rados.rados.0.smithi105.stdout:25: finishing write tid 2 to smithi10548823-25 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:24: finishing write tid 1 to smithi10548823-24 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:24: finishing write tid 2 to smithi10548823-24 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:24: finishing write tid 3 to smithi10548823-24 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:24: finishing write tid 4 to smithi10548823-24 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:24: oid 24 updating version 0 to 10 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:24: oid 24 updating version 10 to 12 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:24: oid 24 version 12 is already newer than 11 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:26: finishing write tid 1 to smithi10548823-26 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:26: finishing write tid 2 to smithi10548823-26 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:26: finishing write tid 3 to smithi10548823-26 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:26: finishing write tid 4 to smithi10548823-26 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:26: oid 26 updating version 0 to 13 2024-04-20T02:50:41.774 INFO:tasks.rados.rados.0.smithi105.stdout:26: oid 26 updating version 13 to 15 2024-04-20T02:50:41.775 INFO:tasks.rados.rados.0.smithi105.stdout:26: oid 26 version 15 is already newer than 14 2024-04-20T02:50:41.775 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 15 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:41.775 INFO:tasks.rados.rados.0.smithi105.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-20T02:50:41.775 INFO:tasks.rados.rados.0.smithi105.stdout:25: finishing write tid 3 to smithi10548823-25 2024-04-20T02:50:41.775 INFO:tasks.rados.rados.0.smithi105.stdout:25: finishing write tid 4 to smithi10548823-25 2024-04-20T02:50:41.775 INFO:tasks.rados.rados.0.smithi105.stdout:25: oid 25 updating version 0 to 9 2024-04-20T02:50:41.775 INFO:tasks.rados.rados.0.smithi105.stdout:25: oid 25 version 9 is already newer than 8 2024-04-20T02:50:41.775 INFO:tasks.rados.rados.0.smithi105.stdout:25: oid 25 version 9 is already newer than 7 2024-04-20T02:50:41.775 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:41.775 INFO:tasks.rados.rados.0.smithi105.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:28: finishing write tid 1 to smithi10548823-28 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:28: finishing write tid 2 to smithi10548823-28 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:28: finishing write tid 3 to smithi10548823-28 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:28: finishing write tid 4 to smithi10548823-28 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:28: oid 28 updating version 0 to 13 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:28: oid 28 updating version 13 to 14 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:28: oid 28 updating version 14 to 15 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:27: finishing write tid 1 to smithi10548823-27 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:27: finishing write tid 2 to smithi10548823-27 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:27: finishing write tid 3 to smithi10548823-27 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:27: finishing write tid 4 to smithi10548823-27 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:27: oid 27 updating version 0 to 11 2024-04-20T02:50:41.776 INFO:tasks.rados.rados.0.smithi105.stdout:27: oid 27 updating version 11 to 12 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:27: oid 27 version 12 is already newer than 10 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:29: finishing write tid 1 to smithi10548823-29 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:29: finishing write tid 2 to smithi10548823-29 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:29: finishing write tid 3 to smithi10548823-29 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:29: finishing write tid 4 to smithi10548823-29 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:29: oid 29 updating version 0 to 1 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:29: oid 29 updating version 1 to 3 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:29: oid 29 version 3 is already newer than 2 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 3 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:31: finishing write tid 1 to smithi10548823-31 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:30: finishing write tid 1 to smithi10548823-30 2024-04-20T02:50:41.777 INFO:tasks.rados.rados.0.smithi105.stdout:30: finishing write tid 2 to smithi10548823-30 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:30: finishing write tid 3 to smithi10548823-30 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:30: finishing write tid 4 to smithi10548823-30 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:30: oid 30 updating version 0 to 1 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:30: oid 30 updating version 1 to 3 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:30: oid 30 version 3 is already newer than 2 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 3 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:32: finishing write tid 1 to smithi10548823-32 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:31: finishing write tid 2 to smithi10548823-31 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:31: finishing write tid 3 to smithi10548823-31 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:31: finishing write tid 4 to smithi10548823-31 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:31: oid 31 updating version 0 to 4 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:31: oid 31 updating version 4 to 5 2024-04-20T02:50:41.778 INFO:tasks.rados.rados.0.smithi105.stdout:31: oid 31 updating version 5 to 6 2024-04-20T02:50:41.779 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:41.779 INFO:tasks.rados.rados.0.smithi105.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-20T02:50:41.779 INFO:tasks.rados.rados.0.smithi105.stdout:32: finishing write tid 2 to smithi10548823-32 2024-04-20T02:50:41.779 INFO:tasks.rados.rados.0.smithi105.stdout:32: finishing write tid 3 to smithi10548823-32 2024-04-20T02:50:41.779 INFO:tasks.rados.rados.0.smithi105.stdout:32: finishing write tid 4 to smithi10548823-32 2024-04-20T02:50:41.779 INFO:tasks.rados.rados.0.smithi105.stdout:32: oid 32 updating version 0 to 3 2024-04-20T02:50:41.779 INFO:tasks.rados.rados.0.smithi105.stdout:32: oid 32 version 3 is already newer than 1 2024-04-20T02:50:41.779 INFO:tasks.rados.rados.0.smithi105.stdout:32: oid 32 version 3 is already newer than 2 2024-04-20T02:50:41.779 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:33: finishing write tid 1 to smithi10548823-33 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:33: finishing write tid 2 to smithi10548823-33 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:33: finishing write tid 3 to smithi10548823-33 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:33: finishing write tid 4 to smithi10548823-33 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:33: oid 33 updating version 0 to 3 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:33: oid 33 version 3 is already newer than 2 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:33: oid 33 version 3 is already newer than 1 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 3 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:34: finishing write tid 1 to smithi10548823-34 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:34: finishing write tid 2 to smithi10548823-34 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:34: finishing write tid 3 to smithi10548823-34 2024-04-20T02:50:41.780 INFO:tasks.rados.rados.0.smithi105.stdout:34: finishing write tid 4 to smithi10548823-34 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:34: oid 34 updating version 0 to 13 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:34: oid 34 updating version 13 to 14 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:34: oid 34 updating version 14 to 15 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:22: done (15 left) 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:23: done (14 left) 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:24: done (13 left) 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:25: done (12 left) 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:26: done (11 left) 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:27: done (10 left) 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:28: done (9 left) 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:29: done (8 left) 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:30: done (7 left) 2024-04-20T02:50:41.781 INFO:tasks.rados.rados.0.smithi105.stdout:31: done (6 left) 2024-04-20T02:50:41.782 INFO:tasks.rados.rados.0.smithi105.stdout:32: done (5 left) 2024-04-20T02:50:41.782 INFO:tasks.rados.rados.0.smithi105.stdout:33: done (4 left) 2024-04-20T02:50:41.782 INFO:tasks.rados.rados.0.smithi105.stdout:34: done (3 left) 2024-04-20T02:50:41.782 INFO:tasks.rados.rados.0.smithi105.stdout:38: write initial oid 38 2024-04-20T02:50:41.782 INFO:tasks.rados.rados.0.smithi105.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-04-20T02:50:41.782 INFO:tasks.rados.rados.0.smithi105.stdout:38: writing smithi10548823-38 from 621859 to 1071484 tid 1 2024-04-20T02:50:41.786 INFO:tasks.rados.rados.0.smithi105.stdout:38: writing smithi10548823-38 from 1865716 to 2621859 tid 2 2024-04-20T02:50:41.786 INFO:tasks.rados.rados.0.smithi105.stdout:39: write initial oid 39 2024-04-20T02:50:41.787 INFO:tasks.rados.rados.0.smithi105.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-04-20T02:50:41.792 INFO:tasks.rados.rados.0.smithi105.stdout:39: writing smithi10548823-39 from 638666 to 1163540 tid 1 2024-04-20T02:50:41.801 INFO:tasks.rados.rados.0.smithi105.stdout:39: writing smithi10548823-39 from 1924198 to 2561203 tid 2 2024-04-20T02:50:41.801 INFO:tasks.rados.rados.0.smithi105.stdout:40: write initial oid 40 2024-04-20T02:50:41.801 INFO:tasks.rados.rados.0.smithi105.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-04-20T02:50:41.803 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:41.803 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:41.808 INFO:tasks.rados.rados.0.smithi105.stdout:40: writing smithi10548823-40 from 655473 to 1371949 tid 1 2024-04-20T02:50:41.815 INFO:tasks.rados.rados.0.smithi105.stdout:40: writing smithi10548823-40 from 2099033 to 2655473 tid 2 2024-04-20T02:50:41.816 INFO:tasks.rados.rados.0.smithi105.stdout:41: write initial oid 41 2024-04-20T02:50:41.816 INFO:tasks.rados.rados.0.smithi105.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-04-20T02:50:41.818 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:50:41.823 INFO:tasks.rados.rados.0.smithi105.stdout:41: writing smithi10548823-41 from 672280 to 1464005 tid 1 2024-04-20T02:50:41.879 INFO:tasks.rados.rados.0.smithi105.stdout:41: writing smithi10548823-41 from 2157515 to 2672280 tid 2 2024-04-20T02:50:41.879 INFO:tasks.rados.rados.0.smithi105.stdout:42: write initial oid 42 2024-04-20T02:50:41.879 INFO:tasks.rados.rados.0.smithi105.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-04-20T02:50:41.879 INFO:tasks.rados.rados.0.smithi105.stdout:42: writing smithi10548823-42 from 689087 to 1156061 tid 1 2024-04-20T02:50:41.879 INFO:tasks.rados.rados.0.smithi105.stdout:42: writing smithi10548823-42 from 1699644 to 2483976 tid 2 2024-04-20T02:50:41.880 INFO:tasks.rados.rados.0.smithi105.stdout:43: write initial oid 43 2024-04-20T02:50:41.880 INFO:tasks.rados.rados.0.smithi105.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-04-20T02:50:41.880 INFO:tasks.rados.rados.0.smithi105.stdout:43: writing smithi10548823-43 from 705894 to 1248117 tid 1 2024-04-20T02:50:41.880 INFO:tasks.rados.rados.0.smithi105.stdout:43: writing smithi10548823-43 from 1758126 to 2402469 tid 2 2024-04-20T02:50:41.880 INFO:tasks.rados.rados.0.smithi105.stdout:44: write initial oid 44 2024-04-20T02:50:41.880 INFO:tasks.rados.rados.0.smithi105.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-04-20T02:50:41.880 INFO:tasks.rados.rados.0.smithi105.stdout:44: writing smithi10548823-44 from 722701 to 1340173 tid 1 2024-04-20T02:50:41.880 INFO:tasks.rados.rados.0.smithi105.stdout:44: writing smithi10548823-44 from 1816608 to 2604609 tid 2 2024-04-20T02:50:41.880 INFO:tasks.rados.rados.0.smithi105.stdout:45: write initial oid 45 2024-04-20T02:50:41.880 INFO:tasks.rados.rados.0.smithi105.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-04-20T02:50:41.881 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:41.879+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:41.885 INFO:tasks.rados.rados.0.smithi105.stdout:45: writing smithi10548823-45 from 739508 to 1432229 tid 1 2024-04-20T02:50:41.891 INFO:tasks.rados.rados.0.smithi105.stdout:45: writing smithi10548823-45 from 1875090 to 2523102 tid 2 2024-04-20T02:50:41.891 INFO:tasks.rados.rados.0.smithi105.stdout:46: write initial oid 46 2024-04-20T02:50:41.891 INFO:tasks.rados.rados.0.smithi105.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-04-20T02:50:41.901 INFO:tasks.rados.rados.0.smithi105.stdout:46: writing smithi10548823-46 from 756315 to 1524285 tid 1 2024-04-20T02:50:41.909 INFO:tasks.rados.rados.0.smithi105.stdout:46: writing smithi10548823-46 from 1933572 to 2725242 tid 2 2024-04-20T02:50:41.909 INFO:tasks.rados.rados.0.smithi105.stdout:47: write initial oid 47 2024-04-20T02:50:41.909 INFO:tasks.rados.rados.0.smithi105.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-04-20T02:50:41.917 INFO:tasks.rados.rados.0.smithi105.stdout:47: writing smithi10548823-47 from 773122 to 1332694 tid 1 2024-04-20T02:50:41.924 INFO:tasks.rados.rados.0.smithi105.stdout:47: writing smithi10548823-47 from 1992054 to 2643735 tid 2 2024-04-20T02:50:41.924 INFO:tasks.rados.rados.0.smithi105.stdout:48: write initial oid 48 2024-04-20T02:50:41.925 INFO:tasks.rados.rados.0.smithi105.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-04-20T02:50:41.932 INFO:tasks.rados.rados.0.smithi105.stdout:48: writing smithi10548823-48 from 789929 to 1424750 tid 1 2024-04-20T02:50:41.942 INFO:tasks.rados.rados.0.smithi105.stdout:48: writing smithi10548823-48 from 2050536 to 2789929 tid 2 2024-04-20T02:50:41.942 INFO:tasks.rados.rados.0.smithi105.stdout:49: write initial oid 49 2024-04-20T02:50:41.942 INFO:tasks.rados.rados.0.smithi105.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-04-20T02:50:41.946 INFO:tasks.rados.rados.0.smithi105.stdout:49: writing smithi10548823-49 from 406736 to 1116806 tid 1 2024-04-20T02:50:41.954 INFO:tasks.rados.rados.0.smithi105.stdout:49: writing smithi10548823-49 from 1709018 to 2364368 tid 2 2024-04-20T02:50:41.954 INFO:tasks.rados.rados.0.smithi105.stdout:50: write initial oid 50 2024-04-20T02:50:41.954 INFO:tasks.rados.rados.0.smithi105.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-04-20T02:50:41.960 INFO:tasks.rados.rados.0.smithi105.stdout:50: writing smithi10548823-50 from 423543 to 1208862 tid 1 2024-04-20T02:50:41.973 INFO:tasks.rados.rados.0.smithi105.stdout:50: writing smithi10548823-50 from 1767500 to 2566508 tid 2 2024-04-20T02:50:41.973 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:41.973 INFO:tasks.rados.rados.0.smithi105.stdout:36: finishing write tid 1 to smithi10548823-36 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:36: finishing write tid 2 to smithi10548823-36 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:36: finishing write tid 3 to smithi10548823-36 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:36: finishing write tid 4 to smithi10548823-36 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:36: oid 36 updating version 0 to 17 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:36: oid 36 version 17 is already newer than 16 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:36: oid 36 updating version 17 to 18 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 18 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:35: finishing write tid 1 to smithi10548823-35 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:35: finishing write tid 2 to smithi10548823-35 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:35: finishing write tid 3 to smithi10548823-35 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:35: finishing write tid 4 to smithi10548823-35 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:35: oid 35 updating version 0 to 1 2024-04-20T02:50:41.974 INFO:tasks.rados.rados.0.smithi105.stdout:35: oid 35 updating version 1 to 3 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:35: oid 35 version 3 is already newer than 2 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 3 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:37: finishing write tid 1 to smithi10548823-37 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:37: finishing write tid 2 to smithi10548823-37 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:37: finishing write tid 3 to smithi10548823-37 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:37: finishing write tid 4 to smithi10548823-37 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:37: oid 37 updating version 0 to 5 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:37: oid 37 version 5 is already newer than 4 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:37: oid 37 updating version 5 to 6 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:39: finishing write tid 1 to smithi10548823-39 2024-04-20T02:50:41.975 INFO:tasks.rados.rados.0.smithi105.stdout:39: finishing write tid 2 to smithi10548823-39 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:39: finishing write tid 3 to smithi10548823-39 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:39: finishing write tid 4 to smithi10548823-39 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:39: oid 39 updating version 0 to 11 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:39: oid 39 updating version 11 to 12 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:39: oid 39 version 12 is already newer than 10 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:40: finishing write tid 1 to smithi10548823-40 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:38: finishing write tid 1 to smithi10548823-38 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:40: finishing write tid 2 to smithi10548823-40 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:40: finishing write tid 3 to smithi10548823-40 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:38: finishing write tid 2 to smithi10548823-38 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:38: finishing write tid 3 to smithi10548823-38 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:38: finishing write tid 4 to smithi10548823-38 2024-04-20T02:50:41.976 INFO:tasks.rados.rados.0.smithi105.stdout:38: oid 38 updating version 0 to 8 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:38: oid 38 version 8 is already newer than 7 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:38: oid 38 updating version 8 to 9 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:40: finishing write tid 4 to smithi10548823-40 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:40: oid 40 updating version 0 to 20 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:40: oid 40 updating version 20 to 21 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:40: oid 40 version 21 is already newer than 19 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 21 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:41: finishing write tid 1 to smithi10548823-41 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:41: finishing write tid 2 to smithi10548823-41 2024-04-20T02:50:41.977 INFO:tasks.rados.rados.0.smithi105.stdout:41: finishing write tid 3 to smithi10548823-41 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:41: finishing write tid 4 to smithi10548823-41 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:41: oid 41 updating version 0 to 18 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:41: oid 41 version 18 is already newer than 16 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:41: oid 41 version 18 is already newer than 17 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 18 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:42: finishing write tid 1 to smithi10548823-42 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:42: finishing write tid 2 to smithi10548823-42 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:42: finishing write tid 3 to smithi10548823-42 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:42: finishing write tid 4 to smithi10548823-42 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:42: oid 42 updating version 0 to 6 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:42: oid 42 version 6 is already newer than 5 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:42: oid 42 version 6 is already newer than 4 2024-04-20T02:50:41.978 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:43: finishing write tid 1 to smithi10548823-43 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:44: finishing write tid 1 to smithi10548823-44 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:43: finishing write tid 2 to smithi10548823-43 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:43: finishing write tid 3 to smithi10548823-43 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:43: finishing write tid 4 to smithi10548823-43 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:43: oid 43 updating version 0 to 15 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:43: oid 43 version 15 is already newer than 14 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:43: oid 43 version 15 is already newer than 13 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 15 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:44: finishing write tid 2 to smithi10548823-44 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:44: finishing write tid 3 to smithi10548823-44 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:44: finishing write tid 4 to smithi10548823-44 2024-04-20T02:50:41.979 INFO:tasks.rados.rados.0.smithi105.stdout:44: oid 44 updating version 0 to 4 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:44: oid 44 updating version 4 to 6 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:44: oid 44 version 6 is already newer than 5 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:45: finishing write tid 1 to smithi10548823-45 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:47: finishing write tid 1 to smithi10548823-47 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:47: finishing write tid 2 to smithi10548823-47 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:47: finishing write tid 3 to smithi10548823-47 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:47: finishing write tid 4 to smithi10548823-47 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:47: oid 47 updating version 0 to 8 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:47: oid 47 updating version 8 to 9 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:47: oid 47 version 9 is already newer than 7 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:35: done (15 left) 2024-04-20T02:50:41.980 INFO:tasks.rados.rados.0.smithi105.stdout:36: done (14 left) 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:37: done (13 left) 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:38: done (12 left) 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:39: done (11 left) 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:40: done (10 left) 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:41: done (9 left) 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:42: done (8 left) 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:43: done (7 left) 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:44: done (6 left) 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:47: done (5 left) 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:51: copy oid 2 target oid 2 low_tier 2024-04-20T02:50:41.981 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 2 to 2 low_tier tgt_pool_name low_tier 2024-04-20T02:50:41.982 INFO:tasks.rados.rados.0.smithi105.stdout:52: copy oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:41.982 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 3 to 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:41.982 INFO:tasks.rados.rados.0.smithi105.stdout:53: copy oid 4 target oid 4 low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 4 to 4 low_tier tgt_pool_name low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:54: copy oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 5 to 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:55: copy oid 6 target oid 6 low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 6 to 6 low_tier tgt_pool_name low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:56: copy oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 7 to 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:57: copy oid 8 target oid 8 low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 8 to 8 low_tier tgt_pool_name low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:58: copy oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 9 to 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:59: copy oid 10 target oid 10 low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 10 to 10 low_tier tgt_pool_name low_tier 2024-04-20T02:50:41.983 INFO:tasks.rados.rados.0.smithi105.stdout:60: copy oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:41.984 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 11 to 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:41.984 INFO:tasks.rados.rados.0.smithi105.stdout:61: copy oid 12 target oid 12 low_tier 2024-04-20T02:50:41.984 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 12 to 12 low_tier tgt_pool_name low_tier 2024-04-20T02:50:41.984 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:41.985 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:41.985+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:41.987 INFO:tasks.rados.rados.0.smithi105.stdout:45: finishing write tid 2 to smithi10548823-45 2024-04-20T02:50:41.987 INFO:tasks.rados.rados.0.smithi105.stdout:45: finishing write tid 3 to smithi10548823-45 2024-04-20T02:50:41.987 INFO:tasks.rados.rados.0.smithi105.stdout:45: finishing write tid 4 to smithi10548823-45 2024-04-20T02:50:41.987 INFO:tasks.rados.rados.0.smithi105.stdout:45: oid 45 updating version 0 to 7 2024-04-20T02:50:41.987 INFO:tasks.rados.rados.0.smithi105.stdout:45: oid 45 updating version 7 to 8 2024-04-20T02:50:41.987 INFO:tasks.rados.rados.0.smithi105.stdout:45: oid 45 updating version 8 to 9 2024-04-20T02:50:41.988 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:41.988 INFO:tasks.rados.rados.0.smithi105.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-20T02:50:41.991 INFO:tasks.rados.rados.0.smithi105.stdout:48: finishing write tid 1 to smithi10548823-48 2024-04-20T02:50:42.002 INFO:tasks.rados.rados.0.smithi105.stdout:45: done (15 left) 2024-04-20T02:50:42.002 INFO:tasks.rados.rados.0.smithi105.stdout:62: copy oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.002 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 13 to 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.002 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.002 INFO:tasks.rados.rados.0.smithi105.stdout:46: finishing write tid 1 to smithi10548823-46 2024-04-20T02:50:42.002 INFO:tasks.rados.rados.0.smithi105.stdout:46: finishing write tid 2 to smithi10548823-46 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout:46: finishing write tid 3 to smithi10548823-46 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout:46: finishing write tid 4 to smithi10548823-46 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout:46: oid 46 updating version 0 to 6 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout:46: oid 46 version 6 is already newer than 4 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout:46: oid 46 version 6 is already newer than 5 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout:46: done (15 left) 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout:63: copy oid 14 target oid 14 low_tier 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 14 to 14 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.003 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.010 INFO:tasks.rados.rados.0.smithi105.stdout:49: finishing write tid 1 to smithi10548823-49 2024-04-20T02:50:42.019 INFO:tasks.rados.rados.0.smithi105.stdout:48: finishing write tid 2 to smithi10548823-48 2024-04-20T02:50:42.019 INFO:tasks.rados.rados.0.smithi105.stdout:48: finishing write tid 3 to smithi10548823-48 2024-04-20T02:50:42.019 INFO:tasks.rados.rados.0.smithi105.stdout:48: finishing write tid 4 to smithi10548823-48 2024-04-20T02:50:42.019 INFO:tasks.rados.rados.0.smithi105.stdout:48: oid 48 updating version 0 to 8 2024-04-20T02:50:42.019 INFO:tasks.rados.rados.0.smithi105.stdout:48: oid 48 version 8 is already newer than 7 2024-04-20T02:50:42.019 INFO:tasks.rados.rados.0.smithi105.stdout:48: oid 48 updating version 8 to 9 2024-04-20T02:50:42.019 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:42.019 INFO:tasks.rados.rados.0.smithi105.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-20T02:50:42.020 INFO:tasks.rados.rados.0.smithi105.stdout:48: done (15 left) 2024-04-20T02:50:42.020 INFO:tasks.rados.rados.0.smithi105.stdout:64: copy oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.021 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 15 to 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.021 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.038 INFO:tasks.rados.rados.0.smithi105.stdout:49: finishing write tid 2 to smithi10548823-49 2024-04-20T02:50:42.038 INFO:tasks.rados.rados.0.smithi105.stdout:49: finishing write tid 3 to smithi10548823-49 2024-04-20T02:50:42.038 INFO:tasks.rados.rados.0.smithi105.stdout:49: finishing write tid 4 to smithi10548823-49 2024-04-20T02:50:42.038 INFO:tasks.rados.rados.0.smithi105.stdout:49: oid 49 updating version 0 to 11 2024-04-20T02:50:42.038 INFO:tasks.rados.rados.0.smithi105.stdout:49: oid 49 updating version 11 to 12 2024-04-20T02:50:42.038 INFO:tasks.rados.rados.0.smithi105.stdout:49: oid 49 version 12 is already newer than 10 2024-04-20T02:50:42.038 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:42.038 INFO:tasks.rados.rados.0.smithi105.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-20T02:50:42.039 INFO:tasks.rados.rados.0.smithi105.stdout:50: finishing write tid 1 to smithi10548823-50 2024-04-20T02:50:42.039 INFO:tasks.rados.rados.0.smithi105.stdout:50: finishing write tid 2 to smithi10548823-50 2024-04-20T02:50:42.039 INFO:tasks.rados.rados.0.smithi105.stdout:49: done (15 left) 2024-04-20T02:50:42.039 INFO:tasks.rados.rados.0.smithi105.stdout:65: copy oid 16 target oid 16 low_tier 2024-04-20T02:50:42.039 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 16 to 16 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.039 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.056 INFO:tasks.rados.rados.0.smithi105.stdout:50: finishing write tid 3 to smithi10548823-50 2024-04-20T02:50:42.056 INFO:tasks.rados.rados.0.smithi105.stdout:50: finishing write tid 4 to smithi10548823-50 2024-04-20T02:50:42.056 INFO:tasks.rados.rados.0.smithi105.stdout:50: oid 50 updating version 0 to 4 2024-04-20T02:50:42.057 INFO:tasks.rados.rados.0.smithi105.stdout:50: oid 50 updating version 4 to 6 2024-04-20T02:50:42.057 INFO:tasks.rados.rados.0.smithi105.stdout:50: oid 50 version 6 is already newer than 5 2024-04-20T02:50:42.057 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 6 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:42.057 INFO:tasks.rados.rados.0.smithi105.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-20T02:50:42.057 INFO:tasks.rados.rados.0.smithi105.stdout:50: done (15 left) 2024-04-20T02:50:42.057 INFO:tasks.rados.rados.0.smithi105.stdout:66: copy oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.057 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 17 to 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.057 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.095 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:42.086+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:42.102 INFO:tasks.rados.rados.0.smithi105.stdout:53: finishing copy op to oid 4 low_tier 2024-04-20T02:50:42.102 INFO:tasks.rados.rados.0.smithi105.stdout:53: done (15 left) 2024-04-20T02:50:42.102 INFO:tasks.rados.rados.0.smithi105.stdout:67: copy oid 18 target oid 18 low_tier 2024-04-20T02:50:42.102 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 18 to 18 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.102 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.116 INFO:tasks.rados.rados.0.smithi105.stdout:60: finishing copy op to oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.120 INFO:tasks.rados.rados.0.smithi105.stdout:60: done (15 left) 2024-04-20T02:50:42.120 INFO:tasks.rados.rados.0.smithi105.stdout:68: copy oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.120 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 19 to 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.120 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.125 INFO:tasks.rados.rados.0.smithi105.stdout:55: finishing copy op to oid 6 low_tier 2024-04-20T02:50:42.125 INFO:tasks.rados.rados.0.smithi105.stdout:56: finishing copy op to oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.125 INFO:tasks.rados.rados.0.smithi105.stdout:55: done (15 left) 2024-04-20T02:50:42.125 INFO:tasks.rados.rados.0.smithi105.stdout:56: done (14 left) 2024-04-20T02:50:42.125 INFO:tasks.rados.rados.0.smithi105.stdout:69: copy oid 20 target oid 20 low_tier 2024-04-20T02:50:42.125 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 20 to 20 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.125 INFO:tasks.rados.rados.0.smithi105.stdout:70: copy oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.126 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 21 to 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.126 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.126 INFO:tasks.rados.rados.0.smithi105.stdout:52: finishing copy op to oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.126 INFO:tasks.rados.rados.0.smithi105.stdout:52: done (15 left) 2024-04-20T02:50:42.126 INFO:tasks.rados.rados.0.smithi105.stdout:71: copy oid 22 target oid 22 low_tier 2024-04-20T02:50:42.126 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 22 to 22 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.126 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.126 INFO:tasks.rados.rados.0.smithi105.stdout:59: finishing copy op to oid 10 low_tier 2024-04-20T02:50:42.126 INFO:tasks.rados.rados.0.smithi105.stdout:59: done (15 left) 2024-04-20T02:50:42.126 INFO:tasks.rados.rados.0.smithi105.stdout:72: copy oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.127 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 23 to 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.127 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.131 INFO:teuthology.orchestra.run.smithi138.stderr:Exporting 4.2 info 4.2( empty local-lis/les=22/23 n=0 ec=22/22 lis/c=22/22 les/c/f=23/23/0 sis=22) 2024-04-20T02:50:42.132 INFO:teuthology.orchestra.run.smithi138.stderr:Export successful 2024-04-20T02:50:42.135 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:42.135 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:42.140 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:50:42.161 INFO:tasks.rados.rados.0.smithi105.stdout:61: finishing copy op to oid 12 low_tier 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:58: finishing copy op to oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:54: finishing copy op to oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:51: finishing copy op to oid 2 low_tier 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:63: finishing copy op to oid 14 low_tier 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:64: finishing copy op to oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:51: done (15 left) 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:54: done (14 left) 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:58: done (13 left) 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:61: done (12 left) 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:63: done (11 left) 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:64: done (10 left) 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:73: copy oid 24 target oid 24 low_tier 2024-04-20T02:50:42.162 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 24 to 24 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout:74: copy oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 25 to 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout:75: copy oid 26 target oid 26 low_tier 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 26 to 26 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout:76: copy oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 27 to 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout:77: copy oid 28 target oid 28 low_tier 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 28 to 28 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout:78: copy oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 29 to 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.163 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.207 INFO:tasks.rados.rados.0.smithi105.stdout:66: finishing copy op to oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.207 INFO:tasks.rados.rados.0.smithi105.stdout:62: finishing copy op to oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.207 INFO:tasks.rados.rados.0.smithi105.stdout:62: done (15 left) 2024-04-20T02:50:42.207 INFO:tasks.rados.rados.0.smithi105.stdout:66: done (14 left) 2024-04-20T02:50:42.207 INFO:tasks.rados.rados.0.smithi105.stdout:79: copy oid 30 target oid 30 low_tier 2024-04-20T02:50:42.207 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 30 to 30 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.207 INFO:tasks.rados.rados.0.smithi105.stdout:80: copy oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.207 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 31 to 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.207 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.213 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:42.206+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:42.247 INFO:tasks.rados.rados.0.smithi105.stdout:72: finishing copy op to oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.247 INFO:tasks.rados.rados.0.smithi105.stdout:72: done (15 left) 2024-04-20T02:50:42.247 INFO:tasks.rados.rados.0.smithi105.stdout:81: copy oid 32 target oid 32 low_tier 2024-04-20T02:50:42.247 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 32 to 32 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.247 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.273 INFO:tasks.rados.rados.0.smithi105.stdout:74: finishing copy op to oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.274 INFO:tasks.rados.rados.0.smithi105.stdout:74: done (15 left) 2024-04-20T02:50:42.274 INFO:tasks.rados.rados.0.smithi105.stdout:82: copy oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.274 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 33 to 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.274 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.277 INFO:tasks.rados.rados.0.smithi105.stdout:68: finishing copy op to oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.277 INFO:tasks.rados.rados.0.smithi105.stdout:68: done (15 left) 2024-04-20T02:50:42.277 INFO:tasks.rados.rados.0.smithi105.stdout:83: copy oid 34 target oid 34 low_tier 2024-04-20T02:50:42.277 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 34 to 34 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.277 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.281 INFO:tasks.rados.rados.0.smithi105.stdout:76: finishing copy op to oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.281 INFO:tasks.rados.rados.0.smithi105.stdout:76: done (15 left) 2024-04-20T02:50:42.281 INFO:tasks.rados.rados.0.smithi105.stdout:84: copy oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.281 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 35 to 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.281 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.281 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:50:42.290 INFO:tasks.rados.rados.0.smithi105.stdout:75: finishing copy op to oid 26 low_tier 2024-04-20T02:50:42.290 INFO:tasks.rados.rados.0.smithi105.stdout:57: finishing copy op to oid 8 low_tier 2024-04-20T02:50:42.290 INFO:tasks.rados.rados.0.smithi105.stdout:57: done (15 left) 2024-04-20T02:50:42.290 INFO:tasks.rados.rados.0.smithi105.stdout:75: done (14 left) 2024-04-20T02:50:42.290 INFO:tasks.rados.rados.0.smithi105.stdout:85: copy oid 36 target oid 36 low_tier 2024-04-20T02:50:42.290 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 36 to 36 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.291 INFO:tasks.rados.rados.0.smithi105.stdout:86: copy oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.291 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 37 to 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.291 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.291 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:42.290+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:65: finishing copy op to oid 16 low_tier 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:71: finishing copy op to oid 22 low_tier 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:65: done (15 left) 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:71: done (14 left) 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:87: copy oid 38 target oid 38 low_tier 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 38 to 38 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:88: copy oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 39 to 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:70: finishing copy op to oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:70: done (15 left) 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:89: copy oid 40 target oid 40 low_tier 2024-04-20T02:50:42.300 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 40 to 40 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.301 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.325 INFO:tasks.rados.rados.0.smithi105.stdout:78: finishing copy op to oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.325 INFO:tasks.rados.rados.0.smithi105.stdout:78: done (15 left) 2024-04-20T02:50:42.326 INFO:tasks.rados.rados.0.smithi105.stdout:90: copy oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.326 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 41 to 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.326 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.338 INFO:tasks.rados.rados.0.smithi105.stdout:83: finishing copy op to oid 34 low_tier 2024-04-20T02:50:42.347 INFO:tasks.rados.rados.0.smithi105.stdout:80: finishing copy op to oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.350 INFO:tasks.rados.rados.0.smithi105.stdout:80: done (15 left) 2024-04-20T02:50:42.350 INFO:tasks.rados.rados.0.smithi105.stdout:83: done (14 left) 2024-04-20T02:50:42.350 INFO:tasks.rados.rados.0.smithi105.stdout:91: copy oid 42 target oid 42 low_tier 2024-04-20T02:50:42.350 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 42 to 42 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.350 INFO:tasks.rados.rados.0.smithi105.stdout:92: copy oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.350 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 43 to 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.350 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.354 INFO:tasks.rados.rados.0.smithi105.stdout:73: finishing copy op to oid 24 low_tier 2024-04-20T02:50:42.361 INFO:tasks.rados.rados.0.smithi105.stdout:81: finishing copy op to oid 32 low_tier 2024-04-20T02:50:42.363 INFO:tasks.rados.rados.0.smithi105.stdout:73: done (15 left) 2024-04-20T02:50:42.363 INFO:tasks.rados.rados.0.smithi105.stdout:81: done (14 left) 2024-04-20T02:50:42.363 INFO:tasks.rados.rados.0.smithi105.stdout:93: copy oid 44 target oid 44 low_tier 2024-04-20T02:50:42.363 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 44 to 44 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.363 INFO:tasks.rados.rados.0.smithi105.stdout:94: copy oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.363 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 45 to 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.363 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.389 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:42.389+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:42.422 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:42.423 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:42.438 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:50:42.469 INFO:tasks.rados.rados.0.smithi105.stdout:77: finishing copy op to oid 28 low_tier 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:88: finishing copy op to oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:87: finishing copy op to oid 38 low_tier 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:77: done (15 left) 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:87: done (14 left) 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:88: done (13 left) 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:95: copy oid 46 target oid 46 low_tier 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 46 to 46 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:96: copy oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 47 to 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:97: copy oid 48 target oid 48 low_tier 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 48 to 48 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.470 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.471 INFO:tasks.rados.rados.0.smithi105.stdout:89: finishing copy op to oid 40 low_tier 2024-04-20T02:50:42.471 INFO:tasks.rados.rados.0.smithi105.stdout:89: done (15 left) 2024-04-20T02:50:42.471 INFO:tasks.rados.rados.0.smithi105.stdout:98: copy oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.471 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 49 to 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.471 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout:67: finishing copy op to oid 18 low_tier 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout:67: done (15 left) 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout:99: copy oid 50 target oid 50 low_tier 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 50 to 50 low_tier tgt_pool_name low_tier 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout:69: finishing copy op to oid 20 low_tier 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout:69: done (15 left) 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout:100: copy oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout:copy op oid 1 to 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.472 INFO:tasks.rados.rados.0.smithi105.stdout:79: finishing copy op to oid 30 low_tier 2024-04-20T02:50:42.473 INFO:tasks.rados.rados.0.smithi105.stdout:79: done (15 left) 2024-04-20T02:50:42.473 INFO:tasks.rados.rados.0.smithi105.stdout:101: set_chunk oid 2 target oid 2 low_tier 2024-04-20T02:50:42.473 INFO:tasks.rados.rados.0.smithi105.stdout:101: set_chunk oid 2 offset: 639488 length: 481792 target oid smithi10548823-2 low_tier offset: 639488 snap -1 2024-04-20T02:50:42.473 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.473 INFO:tasks.rados.rados.0.smithi105.stdout:84: finishing copy op to oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.473 INFO:tasks.rados.rados.0.smithi105.stdout:84: done (15 left) 2024-04-20T02:50:42.473 INFO:tasks.rados.rados.0.smithi105.stdout:102: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.473 INFO:tasks.rados.rados.0.smithi105.stdout:102: set_chunk oid 3 offset: 290304 length: 388096 target oid smithi10548823-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 290304 snap -1 2024-04-20T02:50:42.473 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.515 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:42.513+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:42.576 INFO:tasks.rados.rados.0.smithi105.stdout:98: finishing copy op to oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.576 INFO:tasks.rados.rados.0.smithi105.stdout:98: done (15 left) 2024-04-20T02:50:42.576 INFO:tasks.rados.rados.0.smithi105.stdout:103: set_chunk oid 4 target oid 4 low_tier 2024-04-20T02:50:42.576 INFO:tasks.rados.rados.0.smithi105.stdout:103: set_chunk oid 4 offset: 256512 length: 1762304 target oid smithi10548823-4 low_tier offset: 256512 snap -1 2024-04-20T02:50:42.576 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.576 INFO:tasks.rados.rados.0.smithi105.stdout:90: finishing copy op to oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.576 INFO:tasks.rados.rados.0.smithi105.stdout:90: done (15 left) 2024-04-20T02:50:42.576 INFO:tasks.rados.rados.0.smithi105.stdout:104: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.577 INFO:tasks.rados.rados.0.smithi105.stdout:104: set_chunk oid 5 offset: 104960 length: 965632 target oid smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 104960 snap -1 2024-04-20T02:50:42.577 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.584 INFO:tasks.rados.rados.0.smithi105.stdout:82: finishing copy op to oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.584 INFO:tasks.rados.rados.0.smithi105.stdout:85: finishing copy op to oid 36 low_tier 2024-04-20T02:50:42.584 INFO:tasks.rados.rados.0.smithi105.stdout:86: finishing copy op to oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.584 INFO:tasks.rados.rados.0.smithi105.stdout:91: finishing copy op to oid 42 low_tier 2024-04-20T02:50:42.584 INFO:tasks.rados.rados.0.smithi105.stdout:82: done (15 left) 2024-04-20T02:50:42.584 INFO:tasks.rados.rados.0.smithi105.stdout:85: done (14 left) 2024-04-20T02:50:42.584 INFO:tasks.rados.rados.0.smithi105.stdout:86: done (13 left) 2024-04-20T02:50:42.585 INFO:tasks.rados.rados.0.smithi105.stdout:91: done (12 left) 2024-04-20T02:50:42.585 INFO:tasks.rados.rados.0.smithi105.stdout:105: set_chunk oid 6 target oid 6 low_tier 2024-04-20T02:50:42.585 INFO:tasks.rados.rados.0.smithi105.stdout:105: set_chunk oid 6 offset: 557568 length: 1503744 target oid smithi10548823-6 low_tier offset: 557568 snap -1 2024-04-20T02:50:42.585 INFO:tasks.rados.rados.0.smithi105.stdout:106: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.585 INFO:tasks.rados.rados.0.smithi105.stdout:106: set_chunk oid 7 offset: 36352 length: 434176 target oid smithi10548823-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 36352 snap -1 2024-04-20T02:50:42.585 INFO:tasks.rados.rados.0.smithi105.stdout:107: set_chunk oid 8 target oid 8 low_tier 2024-04-20T02:50:42.585 INFO:tasks.rados.rados.0.smithi105.stdout:107: set_chunk oid 8 offset: 659968 length: 1326080 target oid smithi10548823-8 low_tier offset: 659968 snap -1 2024-04-20T02:50:42.585 INFO:tasks.rados.rados.0.smithi105.stdout:108: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.585 INFO:tasks.rados.rados.0.smithi105.stdout:108: set_chunk oid 9 offset: 195072 length: 1740288 target oid smithi10548823-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 195072 snap -1 2024-04-20T02:50:42.585 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.586 INFO:tasks.rados.rados.0.smithi105.stdout:92: finishing copy op to oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.586 INFO:tasks.rados.rados.0.smithi105.stdout:93: finishing copy op to oid 44 low_tier 2024-04-20T02:50:42.586 INFO:tasks.rados.rados.0.smithi105.stdout:92: done (15 left) 2024-04-20T02:50:42.586 INFO:tasks.rados.rados.0.smithi105.stdout:93: done (14 left) 2024-04-20T02:50:42.586 INFO:tasks.rados.rados.0.smithi105.stdout:109: set_chunk oid 10 target oid 10 low_tier 2024-04-20T02:50:42.586 INFO:tasks.rados.rados.0.smithi105.stdout:109: set_chunk oid 10 offset: 1089024 length: 462336 target oid smithi10548823-10 low_tier offset: 1089024 snap -1 2024-04-20T02:50:42.586 INFO:tasks.rados.rados.0.smithi105.stdout:110: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.586 INFO:tasks.rados.rados.0.smithi105.stdout:110: set_chunk oid 11 offset: 330240 length: 817152 target oid smithi10548823-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 330240 snap -1 2024-04-20T02:50:42.586 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.590 INFO:tasks.rados.rados.0.smithi105.stdout:99: finishing copy op to oid 50 low_tier 2024-04-20T02:50:42.591 INFO:tasks.rados.rados.0.smithi105.stdout:99: done (15 left) 2024-04-20T02:50:42.591 INFO:tasks.rados.rados.0.smithi105.stdout:111: set_chunk oid 12 target oid 12 low_tier 2024-04-20T02:50:42.591 INFO:tasks.rados.rados.0.smithi105.stdout:111: set_chunk oid 12 offset: 27136 length: 1488384 target oid smithi10548823-12 low_tier offset: 27136 snap -1 2024-04-20T02:50:42.591 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.594 INFO:tasks.rados.rados.0.smithi105.stdout:107: finishing set_chunk to oid 8 2024-04-20T02:50:42.594 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:42.594 INFO:tasks.rados.rados.0.smithi105.stdout:107: done (15 left) 2024-04-20T02:50:42.594 INFO:tasks.rados.rados.0.smithi105.stdout:112: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.594 INFO:tasks.rados.rados.0.smithi105.stdout:112: set_chunk oid 13 offset: 1675264 length: 485888 target oid smithi10548823-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1675264 snap -1 2024-04-20T02:50:42.594 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.601 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:42.601+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:42.602 INFO:tasks.rados.rados.0.smithi105.stdout:95: finishing copy op to oid 46 low_tier 2024-04-20T02:50:42.602 INFO:tasks.rados.rados.0.smithi105.stdout:95: done (15 left) 2024-04-20T02:50:42.602 INFO:tasks.rados.rados.0.smithi105.stdout:113: set_chunk oid 14 target oid 14 low_tier 2024-04-20T02:50:42.602 INFO:tasks.rados.rados.0.smithi105.stdout:113: set_chunk oid 14 offset: 193024 length: 1353216 target oid smithi10548823-14 low_tier offset: 193024 snap -1 2024-04-20T02:50:42.602 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.602 INFO:tasks.rados.rados.0.smithi105.stdout:103: finishing set_chunk to oid 4 2024-04-20T02:50:42.602 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:42.602 INFO:tasks.rados.rados.0.smithi105.stdout:103: done (15 left) 2024-04-20T02:50:42.603 INFO:tasks.rados.rados.0.smithi105.stdout:114: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.603 INFO:tasks.rados.rados.0.smithi105.stdout:114: set_chunk oid 15 offset: 389120 length: 586240 target oid smithi10548823-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 389120 snap -1 2024-04-20T02:50:42.603 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.603 INFO:tasks.rados.rados.0.smithi105.stdout:101: finishing set_chunk to oid 2 2024-04-20T02:50:42.603 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:42.603 INFO:tasks.rados.rados.0.smithi105.stdout:101: done (15 left) 2024-04-20T02:50:42.603 INFO:tasks.rados.rados.0.smithi105.stdout:115: set_chunk oid 16 target oid 16 low_tier 2024-04-20T02:50:42.603 INFO:tasks.rados.rados.0.smithi105.stdout:115: set_chunk oid 16 offset: 676864 length: 414208 target oid smithi10548823-16 low_tier offset: 676864 snap -1 2024-04-20T02:50:42.603 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.605 INFO:tasks.rados.rados.0.smithi105.stdout:94: finishing copy op to oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.605 INFO:tasks.rados.rados.0.smithi105.stdout:94: done (15 left) 2024-04-20T02:50:42.605 INFO:tasks.rados.rados.0.smithi105.stdout:116: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.605 INFO:tasks.rados.rados.0.smithi105.stdout:116: set_chunk oid 17 offset: 1465344 length: 468480 target oid smithi10548823-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1465344 snap -1 2024-04-20T02:50:42.605 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.605 INFO:tasks.rados.rados.0.smithi105.stdout:106: finishing set_chunk to oid 7 2024-04-20T02:50:42.605 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 9 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:42.606 INFO:tasks.rados.rados.0.smithi105.stdout:106: done (15 left) 2024-04-20T02:50:42.606 INFO:tasks.rados.rados.0.smithi105.stdout:117: set_chunk oid 18 target oid 18 low_tier 2024-04-20T02:50:42.606 INFO:tasks.rados.rados.0.smithi105.stdout:117: set_chunk oid 18 offset: 39936 length: 800256 target oid smithi10548823-18 low_tier offset: 39936 snap -1 2024-04-20T02:50:42.606 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.606 INFO:tasks.rados.rados.0.smithi105.stdout:100: finishing copy op to oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.606 INFO:tasks.rados.rados.0.smithi105.stdout:100: done (15 left) 2024-04-20T02:50:42.606 INFO:tasks.rados.rados.0.smithi105.stdout:118: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.606 INFO:tasks.rados.rados.0.smithi105.stdout:118: set_chunk oid 19 offset: 1462272 length: 73216 target oid smithi10548823-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1462272 snap -1 2024-04-20T02:50:42.606 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.607 INFO:tasks.rados.rados.0.smithi105.stdout:104: finishing set_chunk to oid 5 2024-04-20T02:50:42.607 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:42.607 INFO:tasks.rados.rados.0.smithi105.stdout:104: done (15 left) 2024-04-20T02:50:42.607 INFO:tasks.rados.rados.0.smithi105.stdout:119: set_chunk oid 20 target oid 20 low_tier 2024-04-20T02:50:42.607 INFO:tasks.rados.rados.0.smithi105.stdout:119: set_chunk oid 20 offset: 400384 length: 444928 target oid smithi10548823-20 low_tier offset: 400384 snap -1 2024-04-20T02:50:42.607 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.608 INFO:tasks.rados.rados.0.smithi105.stdout:114: finishing set_chunk to oid 15 2024-04-20T02:50:42.608 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:42.608 INFO:tasks.rados.rados.0.smithi105.stdout:114: done (15 left) 2024-04-20T02:50:42.608 INFO:tasks.rados.rados.0.smithi105.stdout:120: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.608 INFO:tasks.rados.rados.0.smithi105.stdout:120: set_chunk oid 21 offset: 220672 length: 480256 target oid smithi10548823-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 220672 snap -1 2024-04-20T02:50:42.608 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.610 INFO:tasks.rados.rados.0.smithi105.stdout:117: finishing set_chunk to oid 18 2024-04-20T02:50:42.610 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:42.610 INFO:tasks.rados.rados.0.smithi105.stdout:110: finishing set_chunk to oid 11 2024-04-20T02:50:42.610 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:42.611 INFO:tasks.rados.rados.0.smithi105.stdout:110: done (15 left) 2024-04-20T02:50:42.611 INFO:tasks.rados.rados.0.smithi105.stdout:117: done (14 left) 2024-04-20T02:50:42.611 INFO:tasks.rados.rados.0.smithi105.stdout:121: set_chunk oid 22 target oid 22 low_tier 2024-04-20T02:50:42.611 INFO:tasks.rados.rados.0.smithi105.stdout:121: set_chunk oid 22 offset: 542208 length: 592896 target oid smithi10548823-22 low_tier offset: 542208 snap -1 2024-04-20T02:50:42.611 INFO:tasks.rados.rados.0.smithi105.stdout:122: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.611 INFO:tasks.rados.rados.0.smithi105.stdout:122: set_chunk oid 23 offset: 398848 length: 552448 target oid smithi10548823-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 398848 snap -1 2024-04-20T02:50:42.611 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.612 INFO:tasks.rados.rados.0.smithi105.stdout:113: finishing set_chunk to oid 14 2024-04-20T02:50:42.612 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:42.612 INFO:tasks.rados.rados.0.smithi105.stdout:113: done (15 left) 2024-04-20T02:50:42.612 INFO:tasks.rados.rados.0.smithi105.stdout:123: set_chunk oid 24 target oid 24 low_tier 2024-04-20T02:50:42.612 INFO:tasks.rados.rados.0.smithi105.stdout:123: set_chunk oid 24 offset: 1290240 length: 272384 target oid smithi10548823-24 low_tier offset: 1290240 snap -1 2024-04-20T02:50:42.612 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.612 INFO:tasks.rados.rados.0.smithi105.stdout:96: finishing copy op to oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.613 INFO:tasks.rados.rados.0.smithi105.stdout:96: done (15 left) 2024-04-20T02:50:42.613 INFO:tasks.rados.rados.0.smithi105.stdout:124: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.613 INFO:tasks.rados.rados.0.smithi105.stdout:124: set_chunk oid 25 offset: 497152 length: 1769472 target oid smithi10548823-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 497152 snap -1 2024-04-20T02:50:42.613 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.615 INFO:tasks.rados.rados.0.smithi105.stdout:108: finishing set_chunk to oid 9 2024-04-20T02:50:42.615 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 12 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:42.615 INFO:tasks.rados.rados.0.smithi105.stdout:108: done (15 left) 2024-04-20T02:50:42.615 INFO:tasks.rados.rados.0.smithi105.stdout:125: set_chunk oid 26 target oid 26 low_tier 2024-04-20T02:50:42.615 INFO:tasks.rados.rados.0.smithi105.stdout:125: set_chunk oid 26 offset: 2197504 length: 202752 target oid smithi10548823-26 low_tier offset: 2197504 snap -1 2024-04-20T02:50:42.615 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.615 INFO:tasks.rados.rados.0.smithi105.stdout:97: finishing copy op to oid 48 low_tier 2024-04-20T02:50:42.615 INFO:tasks.rados.rados.0.smithi105.stdout:97: done (15 left) 2024-04-20T02:50:42.615 INFO:tasks.rados.rados.0.smithi105.stdout:126: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.615 INFO:tasks.rados.rados.0.smithi105.stdout:126: set_chunk oid 27 offset: 595968 length: 878080 target oid smithi10548823-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 595968 snap -1 2024-04-20T02:50:42.616 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.616 INFO:tasks.rados.rados.0.smithi105.stdout:112: finishing set_chunk to oid 13 2024-04-20T02:50:42.616 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:42.616 INFO:tasks.rados.rados.0.smithi105.stdout:112: done (15 left) 2024-04-20T02:50:42.616 INFO:tasks.rados.rados.0.smithi105.stdout:127: set_chunk oid 28 target oid 28 low_tier 2024-04-20T02:50:42.616 INFO:tasks.rados.rados.0.smithi105.stdout:127: set_chunk oid 28 offset: 939008 length: 829952 target oid smithi10548823-28 low_tier offset: 939008 snap -1 2024-04-20T02:50:42.616 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.619 INFO:tasks.rados.rados.0.smithi105.stdout:120: finishing set_chunk to oid 21 2024-04-20T02:50:42.619 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:42.619 INFO:tasks.rados.rados.0.smithi105.stdout:120: done (15 left) 2024-04-20T02:50:42.619 INFO:tasks.rados.rados.0.smithi105.stdout:128: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.619 INFO:tasks.rados.rados.0.smithi105.stdout:128: set_chunk oid 29 offset: 83968 length: 2255360 target oid smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 83968 snap -1 2024-04-20T02:50:42.619 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.619 INFO:tasks.rados.rados.0.smithi105.stdout:102: finishing set_chunk to oid 3 2024-04-20T02:50:42.619 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:42.620 INFO:tasks.rados.rados.0.smithi105.stdout:102: done (15 left) 2024-04-20T02:50:42.620 INFO:tasks.rados.rados.0.smithi105.stdout:129: set_chunk oid 30 target oid 30 low_tier 2024-04-20T02:50:42.620 INFO:tasks.rados.rados.0.smithi105.stdout:129: set_chunk oid 30 offset: 1668096 length: 719360 target oid smithi10548823-30 low_tier offset: 1668096 snap -1 2024-04-20T02:50:42.620 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.622 INFO:tasks.rados.rados.0.smithi105.stdout:111: finishing set_chunk to oid 12 2024-04-20T02:50:42.622 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:42.622 INFO:tasks.rados.rados.0.smithi105.stdout:111: done (15 left) 2024-04-20T02:50:42.622 INFO:tasks.rados.rados.0.smithi105.stdout:130: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.622 INFO:tasks.rados.rados.0.smithi105.stdout:130: set_chunk oid 31 offset: 940544 length: 754688 target oid smithi10548823-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 940544 snap -1 2024-04-20T02:50:42.622 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.622 INFO:tasks.rados.rados.0.smithi105.stdout:115: finishing set_chunk to oid 16 2024-04-20T02:50:42.622 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 9 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:42.622 INFO:tasks.rados.rados.0.smithi105.stdout:115: done (15 left) 2024-04-20T02:50:42.623 INFO:tasks.rados.rados.0.smithi105.stdout:131: set_chunk oid 32 target oid 32 low_tier 2024-04-20T02:50:42.623 INFO:tasks.rados.rados.0.smithi105.stdout:131: set_chunk oid 32 offset: 820224 length: 1506816 target oid smithi10548823-32 low_tier offset: 820224 snap -1 2024-04-20T02:50:42.623 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:121: finishing set_chunk to oid 22 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:121: done (15 left) 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:132: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:132: set_chunk oid 33 offset: 46592 length: 1990656 target oid smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 46592 snap -1 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:116: finishing set_chunk to oid 17 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:119: finishing set_chunk to oid 20 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:116: done (15 left) 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:119: done (14 left) 2024-04-20T02:50:42.627 INFO:tasks.rados.rados.0.smithi105.stdout:133: set_chunk oid 34 target oid 34 low_tier 2024-04-20T02:50:42.628 INFO:tasks.rados.rados.0.smithi105.stdout:133: set_chunk oid 34 offset: 935936 length: 1004544 target oid smithi10548823-34 low_tier offset: 935936 snap -1 2024-04-20T02:50:42.628 INFO:tasks.rados.rados.0.smithi105.stdout:134: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.628 INFO:tasks.rados.rados.0.smithi105.stdout:134: set_chunk oid 35 offset: 1395712 length: 739840 target oid smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1395712 snap -1 2024-04-20T02:50:42.628 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.629 INFO:tasks.rados.rados.0.smithi105.stdout:105: finishing set_chunk to oid 6 2024-04-20T02:50:42.629 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:42.629 INFO:tasks.rados.rados.0.smithi105.stdout:105: done (15 left) 2024-04-20T02:50:42.629 INFO:tasks.rados.rados.0.smithi105.stdout:135: set_chunk oid 36 target oid 36 low_tier 2024-04-20T02:50:42.629 INFO:tasks.rados.rados.0.smithi105.stdout:135: set_chunk oid 36 offset: 187904 length: 1546240 target oid smithi10548823-36 low_tier offset: 187904 snap -1 2024-04-20T02:50:42.629 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.629 INFO:tasks.rados.rados.0.smithi105.stdout:123: finishing set_chunk to oid 24 2024-04-20T02:50:42.630 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:42.630 INFO:tasks.rados.rados.0.smithi105.stdout:123: done (15 left) 2024-04-20T02:50:42.630 INFO:tasks.rados.rados.0.smithi105.stdout:136: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.630 INFO:tasks.rados.rados.0.smithi105.stdout:136: set_chunk oid 37 offset: 307200 length: 573952 target oid smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 307200 snap -1 2024-04-20T02:50:42.630 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.631 INFO:tasks.rados.rados.0.smithi105.stdout:128: finishing set_chunk to oid 29 2024-04-20T02:50:42.631 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 3 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:42.631 INFO:tasks.rados.rados.0.smithi105.stdout:128: done (15 left) 2024-04-20T02:50:42.631 INFO:tasks.rados.rados.0.smithi105.stdout:137: set_chunk oid 38 target oid 38 low_tier 2024-04-20T02:50:42.631 INFO:tasks.rados.rados.0.smithi105.stdout:137: set_chunk oid 38 offset: 928256 length: 1102336 target oid smithi10548823-38 low_tier offset: 928256 snap -1 2024-04-20T02:50:42.631 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout:109: finishing set_chunk to oid 10 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout:109: done (15 left) 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout:138: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout:138: set_chunk oid 39 offset: 865792 length: 1120768 target oid smithi10548823-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 865792 snap -1 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout:118: finishing set_chunk to oid 19 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 12 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout:118: done (15 left) 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout:139: set_chunk oid 40 target oid 40 low_tier 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout:139: set_chunk oid 40 offset: 499200 length: 127488 target oid smithi10548823-40 low_tier offset: 499200 snap -1 2024-04-20T02:50:42.632 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.633 INFO:tasks.rados.rados.0.smithi105.stdout:131: finishing set_chunk to oid 32 2024-04-20T02:50:42.633 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:42.633 INFO:tasks.rados.rados.0.smithi105.stdout:131: done (15 left) 2024-04-20T02:50:42.633 INFO:tasks.rados.rados.0.smithi105.stdout:140: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.634 INFO:tasks.rados.rados.0.smithi105.stdout:140: set_chunk oid 41 offset: 552448 length: 1430528 target oid smithi10548823-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 552448 snap -1 2024-04-20T02:50:42.634 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:122: finishing set_chunk to oid 23 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:129: finishing set_chunk to oid 30 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 3 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:127: finishing set_chunk to oid 28 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:124: finishing set_chunk to oid 25 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:122: done (15 left) 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:124: done (14 left) 2024-04-20T02:50:42.635 INFO:tasks.rados.rados.0.smithi105.stdout:127: done (13 left) 2024-04-20T02:50:42.636 INFO:tasks.rados.rados.0.smithi105.stdout:129: done (12 left) 2024-04-20T02:50:42.636 INFO:tasks.rados.rados.0.smithi105.stdout:141: set_chunk oid 42 target oid 42 low_tier 2024-04-20T02:50:42.636 INFO:tasks.rados.rados.0.smithi105.stdout:141: set_chunk oid 42 offset: 46592 length: 2137088 target oid smithi10548823-42 low_tier offset: 46592 snap -1 2024-04-20T02:50:42.636 INFO:tasks.rados.rados.0.smithi105.stdout:142: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.636 INFO:tasks.rados.rados.0.smithi105.stdout:142: set_chunk oid 43 offset: 1340416 length: 373248 target oid smithi10548823-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1340416 snap -1 2024-04-20T02:50:42.636 INFO:tasks.rados.rados.0.smithi105.stdout:143: set_chunk oid 44 target oid 44 low_tier 2024-04-20T02:50:42.636 INFO:tasks.rados.rados.0.smithi105.stdout:143: set_chunk oid 44 offset: 170496 length: 864768 target oid smithi10548823-44 low_tier offset: 170496 snap -1 2024-04-20T02:50:42.636 INFO:tasks.rados.rados.0.smithi105.stdout:144: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.636 INFO:tasks.rados.rados.0.smithi105.stdout:144: set_chunk oid 45 offset: 843264 length: 1586688 target oid smithi10548823-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 843264 snap -1 2024-04-20T02:50:42.636 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.637 INFO:tasks.rados.rados.0.smithi105.stdout:130: finishing set_chunk to oid 31 2024-04-20T02:50:42.637 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:42.637 INFO:tasks.rados.rados.0.smithi105.stdout:132: finishing set_chunk to oid 33 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 3 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:130: done (15 left) 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:132: done (14 left) 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:145: set_chunk oid 46 target oid 46 low_tier 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:145: set_chunk oid 46 offset: 636928 length: 889344 target oid smithi10548823-46 low_tier offset: 636928 snap -1 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:146: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:146: set_chunk oid 47 offset: 331264 length: 1913856 target oid smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 331264 snap -1 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:125: finishing set_chunk to oid 26 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 15 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:125: done (15 left) 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:147: set_chunk oid 48 target oid 48 low_tier 2024-04-20T02:50:42.638 INFO:tasks.rados.rados.0.smithi105.stdout:147: set_chunk oid 48 offset: 1618432 length: 529408 target oid smithi10548823-48 low_tier offset: 1618432 snap -1 2024-04-20T02:50:42.639 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout:135: finishing set_chunk to oid 36 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 18 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout:135: done (15 left) 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout:148: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout:148: set_chunk oid 49 offset: 1478144 length: 306688 target oid smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1478144 snap -1 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout:134: finishing set_chunk to oid 35 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 3 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout:134: done (15 left) 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout:149: set_chunk oid 50 target oid 50 low_tier 2024-04-20T02:50:42.640 INFO:tasks.rados.rados.0.smithi105.stdout:149: set_chunk oid 50 offset: 1958400 length: 59904 target oid smithi10548823-50 low_tier offset: 1958400 snap -1 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout:126: finishing set_chunk to oid 27 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout:126: done (15 left) 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout:150: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout:150: set_chunk oid 1 offset: 504832 length: 1271808 target oid smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 504832 snap -1 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout:136: finishing set_chunk to oid 37 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout:136: done (15 left) 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout:151: set_chunk oid 2 target oid 2 low_tier 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout:151: set_chunk oid 2 offset: 1508864 length: 340480 target oid smithi10548823-2 low_tier offset: 1508864 snap -1 2024-04-20T02:50:42.641 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.644 INFO:tasks.rados.rados.0.smithi105.stdout:141: finishing set_chunk to oid 42 2024-04-20T02:50:42.644 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:42.644 INFO:tasks.rados.rados.0.smithi105.stdout:141: done (15 left) 2024-04-20T02:50:42.644 INFO:tasks.rados.rados.0.smithi105.stdout:152: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.644 INFO:tasks.rados.rados.0.smithi105.stdout:152: set_chunk oid 3 offset: 95744 length: 971776 target oid smithi10548823-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 95744 snap -1 2024-04-20T02:50:42.644 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.644 INFO:tasks.rados.rados.0.smithi105.stdout:152: finishing set_chunk to oid 3 2024-04-20T02:50:42.644 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 95744 length: 971776 2024-04-20T02:50:42.644 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:42.644 INFO:tasks.rados.rados.0.smithi105.stdout:144: finishing set_chunk to oid 45 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:144: done (15 left) 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:152: done (14 left) 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:153: set_chunk oid 4 target oid 4 low_tier 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:153: set_chunk oid 4 offset: 1597952 length: 188416 target oid smithi10548823-4 low_tier offset: 1597952 snap -1 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:154: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:154: set_chunk oid 5 offset: 268288 length: 349696 target oid smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 268288 snap -1 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:154: finishing set_chunk to oid 5 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 268288 length: 349696 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:133: finishing set_chunk to oid 34 2024-04-20T02:50:42.645 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:133: done (15 left) 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:154: done (14 left) 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:155: set_chunk oid 6 target oid 6 low_tier 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:155: set_chunk oid 6 offset: 729088 length: 1310720 target oid smithi10548823-6 low_tier offset: 729088 snap -1 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:156: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:156: set_chunk oid 7 offset: 732160 length: 940544 target oid smithi10548823-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 732160 snap -1 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:153: finishing set_chunk to oid 4 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1597952 length: 188416 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:153: done (15 left) 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:157: set_chunk oid 8 target oid 8 low_tier 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout:157: set_chunk oid 8 offset: 1564160 length: 343040 target oid smithi10548823-8 low_tier offset: 1564160 snap -1 2024-04-20T02:50:42.646 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:155: finishing set_chunk to oid 6 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 729088 length: 1310720 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:155: done (15 left) 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:158: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:158: set_chunk oid 9 offset: 868352 length: 240640 target oid smithi10548823-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 868352 snap -1 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:157: finishing set_chunk to oid 8 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1564160 length: 343040 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:157: done (15 left) 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:159: set_chunk oid 10 target oid 10 low_tier 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:159: set_chunk oid 10 offset: 929792 length: 1184768 target oid smithi10548823-10 low_tier offset: 929792 snap -1 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.647 INFO:tasks.rados.rados.0.smithi105.stdout:158: finishing set_chunk to oid 9 2024-04-20T02:50:42.648 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 868352 length: 240640 2024-04-20T02:50:42.648 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:42.648 INFO:tasks.rados.rados.0.smithi105.stdout:158: done (15 left) 2024-04-20T02:50:42.648 INFO:tasks.rados.rados.0.smithi105.stdout:160: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.648 INFO:tasks.rados.rados.0.smithi105.stdout:160: set_chunk oid 11 offset: 225792 length: 1648128 target oid smithi10548823-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 225792 snap -1 2024-04-20T02:50:42.648 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.653 INFO:tasks.rados.rados.0.smithi105.stdout:159: finishing set_chunk to oid 10 2024-04-20T02:50:42.653 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 929792 length: 1184768 2024-04-20T02:50:42.653 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:42.653 INFO:tasks.rados.rados.0.smithi105.stdout:159: done (15 left) 2024-04-20T02:50:42.653 INFO:tasks.rados.rados.0.smithi105.stdout:161: set_chunk oid 12 target oid 12 low_tier 2024-04-20T02:50:42.653 INFO:tasks.rados.rados.0.smithi105.stdout:161: set_chunk oid 12 offset: 610304 length: 989696 target oid smithi10548823-12 low_tier offset: 610304 snap -1 2024-04-20T02:50:42.653 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.653 INFO:tasks.rados.rados.0.smithi105.stdout:137: finishing set_chunk to oid 38 2024-04-20T02:50:42.653 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:42.653 INFO:tasks.rados.rados.0.smithi105.stdout:161: finishing set_chunk to oid 12 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 610304 length: 989696 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:146: finishing set_chunk to oid 47 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:139: finishing set_chunk to oid 40 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 21 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:137: done (15 left) 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:139: done (14 left) 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:146: done (13 left) 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:161: done (12 left) 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:162: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:162: set_chunk oid 13 offset: 1242624 length: 56320 target oid smithi10548823-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1242624 snap -1 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:163: set_chunk oid 14 target oid 14 low_tier 2024-04-20T02:50:42.654 INFO:tasks.rados.rados.0.smithi105.stdout:163: set_chunk oid 14 offset: 736256 length: 484864 target oid smithi10548823-14 low_tier offset: 736256 snap -1 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:164: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:164: set_chunk oid 15 offset: 1457152 length: 204800 target oid smithi10548823-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1457152 snap -1 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:165: set_chunk oid 16 target oid 16 low_tier 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:165: set_chunk oid 16 offset: 973824 length: 340480 target oid smithi10548823-16 low_tier offset: 973824 snap -1 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:165: finishing set_chunk to oid 16 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 973824 length: 340480 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:165: done (15 left) 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:166: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:166: set_chunk oid 17 offset: 1035776 length: 518144 target oid smithi10548823-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1035776 snap -1 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.655 INFO:tasks.rados.rados.0.smithi105.stdout:166: finishing set_chunk to oid 17 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1035776 length: 518144 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:166: done (15 left) 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:167: set_chunk oid 18 target oid 18 low_tier 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:167: set_chunk oid 18 offset: 281600 length: 1565184 target oid smithi10548823-18 low_tier offset: 281600 snap -1 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:167: finishing set_chunk to oid 18 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 281600 length: 1565184 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:167: done (15 left) 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:168: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:168: set_chunk oid 19 offset: 131584 length: 837632 target oid smithi10548823-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 131584 snap -1 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.656 INFO:tasks.rados.rados.0.smithi105.stdout:148: finishing set_chunk to oid 49 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:148: done (15 left) 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:169: set_chunk oid 20 target oid 20 low_tier 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:169: set_chunk oid 20 offset: 390656 length: 1588736 target oid smithi10548823-20 low_tier offset: 390656 snap -1 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:169: finishing set_chunk to oid 20 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 390656 length: 1588736 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:169: done (15 left) 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:170: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:170: set_chunk oid 21 offset: 1272832 length: 834560 target oid smithi10548823-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1272832 snap -1 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:145: finishing set_chunk to oid 46 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:42.657 INFO:tasks.rados.rados.0.smithi105.stdout:145: done (15 left) 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:171: set_chunk oid 22 target oid 22 low_tier 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:171: set_chunk oid 22 offset: 233472 length: 1937920 target oid smithi10548823-22 low_tier offset: 233472 snap -1 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:171: finishing set_chunk to oid 22 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 233472 length: 1937920 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:171: done (15 left) 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:172: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:172: set_chunk oid 23 offset: 170496 length: 664064 target oid smithi10548823-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 170496 snap -1 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:172: finishing set_chunk to oid 23 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 170496 length: 664064 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:172: done (15 left) 2024-04-20T02:50:42.658 INFO:tasks.rados.rados.0.smithi105.stdout:173: set_chunk oid 24 target oid 24 low_tier 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:173: set_chunk oid 24 offset: 874496 length: 1438720 target oid smithi10548823-24 low_tier offset: 874496 snap -1 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:173: finishing set_chunk to oid 24 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 874496 length: 1438720 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:173: done (15 left) 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:174: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:174: set_chunk oid 25 offset: 679424 length: 219136 target oid smithi10548823-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 679424 snap -1 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:143: finishing set_chunk to oid 44 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:149: finishing set_chunk to oid 50 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 6 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:143: done (15 left) 2024-04-20T02:50:42.659 INFO:tasks.rados.rados.0.smithi105.stdout:149: done (14 left) 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:175: set_chunk oid 26 target oid 26 low_tier 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:175: set_chunk oid 26 offset: 859136 length: 1347072 target oid smithi10548823-26 low_tier offset: 859136 snap -1 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:176: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:176: set_chunk oid 27 offset: 3584 length: 209408 target oid smithi10548823-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 3584 snap -1 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:174: finishing set_chunk to oid 25 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 679424 length: 219136 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:174: done (15 left) 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:177: set_chunk oid 28 target oid 28 low_tier 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:177: set_chunk oid 28 offset: 1373184 length: 57344 target oid smithi10548823-28 low_tier offset: 1373184 snap -1 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:175: finishing set_chunk to oid 26 2024-04-20T02:50:42.660 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 859136 length: 1347072 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:175: done (15 left) 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:178: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:178: set_chunk oid 29 offset: 599040 length: 6144 target oid smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 599040 snap -1 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:177: finishing set_chunk to oid 28 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1373184 length: 57344 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:177: done (15 left) 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:179: set_chunk oid 30 target oid 30 low_tier 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:179: set_chunk oid 30 offset: 1115648 length: 374784 target oid smithi10548823-30 low_tier offset: 1115648 snap -1 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:178: finishing set_chunk to oid 29 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 599040 length: 6144 2024-04-20T02:50:42.661 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:160: finishing set_chunk to oid 11 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 225792 length: 1648128 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:160: done (15 left) 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:178: done (14 left) 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:180: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:180: set_chunk oid 31 offset: 232448 length: 779264 target oid smithi10548823-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 232448 snap -1 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:181: set_chunk oid 32 target oid 32 low_tier 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:181: set_chunk oid 32 offset: 859648 length: 139776 target oid smithi10548823-32 low_tier offset: 859648 snap -1 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:180: finishing set_chunk to oid 31 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 232448 length: 779264 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:42.662 INFO:tasks.rados.rados.0.smithi105.stdout:180: done (15 left) 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:182: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:182: set_chunk oid 33 offset: 1404416 length: 81920 target oid smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1404416 snap -1 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:182: finishing set_chunk to oid 33 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1404416 length: 81920 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:182: done (15 left) 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:183: set_chunk oid 34 target oid 34 low_tier 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:183: set_chunk oid 34 offset: 1242112 length: 326144 target oid smithi10548823-34 low_tier offset: 1242112 snap -1 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:183: finishing set_chunk to oid 34 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1242112 length: 326144 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:42.663 INFO:tasks.rados.rados.0.smithi105.stdout:183: done (15 left) 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:184: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:184: set_chunk oid 35 offset: 886784 length: 1444864 target oid smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 886784 snap -1 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:147: finishing set_chunk to oid 48 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:147: done (15 left) 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:185: set_chunk oid 36 target oid 36 low_tier 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:185: set_chunk oid 36 offset: 905728 length: 1409024 target oid smithi10548823-36 low_tier offset: 905728 snap -1 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:185: finishing set_chunk to oid 36 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 905728 length: 1409024 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:140: finishing set_chunk to oid 41 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 18 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:42.664 INFO:tasks.rados.rados.0.smithi105.stdout:140: done (15 left) 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:185: done (14 left) 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:186: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:186: set_chunk oid 37 offset: 653824 length: 1439744 target oid smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 653824 snap -1 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:187: set_chunk oid 38 target oid 38 low_tier 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:187: set_chunk oid 38 offset: 1230336 length: 881152 target oid smithi10548823-38 low_tier offset: 1230336 snap -1 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:150: finishing set_chunk to oid 1 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:150: done (15 left) 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:187 retry set_chunk ! 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 15 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:168: finishing set_chunk to oid 19 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 12 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:42.665 INFO:tasks.rados.rados.0.smithi105.stdout:168: done (14 left) 2024-04-20T02:50:42.666 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 14 2024-04-20T02:50:42.667 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 4.2 2024-04-20T02:50:42.679 INFO:tasks.rados.rados.0.smithi105.stdout:163: finishing set_chunk to oid 14 2024-04-20T02:50:42.679 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 736256 length: 484864 2024-04-20T02:50:42.679 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:42.679 INFO:tasks.rados.rados.0.smithi105.stdout:181: finishing set_chunk to oid 32 2024-04-20T02:50:42.679 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 859648 length: 139776 2024-04-20T02:50:42.679 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:42.679 INFO:tasks.rados.rados.0.smithi105.stdout:184: finishing set_chunk to oid 35 2024-04-20T02:50:42.679 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 886784 length: 1444864 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:186: finishing set_chunk to oid 37 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 653824 length: 1439744 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:187: finishing set_chunk to oid 38 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 1230336 length: 881152 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:138: finishing set_chunk to oid 39 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:142: finishing set_chunk to oid 43 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 15 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:138: done (13 left) 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:142: done (12 left) 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:163: done (11 left) 2024-04-20T02:50:42.680 INFO:tasks.rados.rados.0.smithi105.stdout:181: done (10 left) 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout:184: done (9 left) 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout:186: done (8 left) 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout:187: done (7 left) 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 7 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout:156: finishing set_chunk to oid 7 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 9 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout:156: done (6 left) 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 6 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout:151: finishing set_chunk to oid 2 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout:151: done (5 left) 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 5 2024-04-20T02:50:42.681 INFO:tasks.rados.rados.0.smithi105.stdout:176: finishing set_chunk to oid 27 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:176: done (4 left) 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 4 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:179: finishing set_chunk to oid 30 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 3 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:179: done (3 left) 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 3 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:164: finishing set_chunk to oid 15 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:164: done (2 left) 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 2 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:162: finishing set_chunk to oid 13 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:42.682 INFO:tasks.rados.rados.0.smithi105.stdout:162: done (1 left) 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 1 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:170: finishing set_chunk to oid 21 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:170: done (0 left) 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:188: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:188: set_chunk oid 39 offset: 398848 length: 704512 target oid smithi10548823-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 398848 snap -1 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:189: set_chunk oid 40 target oid 40 low_tier 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:189: set_chunk oid 40 offset: 1293312 length: 572928 target oid smithi10548823-40 low_tier offset: 1293312 snap -1 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:190: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:190: set_chunk oid 41 offset: 1218048 length: 729600 target oid smithi10548823-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1218048 snap -1 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:191: set_chunk oid 42 target oid 42 low_tier 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:191: set_chunk oid 42 offset: 486400 length: 218112 target oid smithi10548823-42 low_tier offset: 486400 snap -1 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:192: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.683 INFO:tasks.rados.rados.0.smithi105.stdout:192: set_chunk oid 43 offset: 181248 length: 1238016 target oid smithi10548823-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 181248 snap -1 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:193: set_chunk oid 44 target oid 44 low_tier 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:193: set_chunk oid 44 offset: 179712 length: 1415168 target oid smithi10548823-44 low_tier offset: 179712 snap -1 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:194: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:194: set_chunk oid 45 offset: 447488 length: 1222656 target oid smithi10548823-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 447488 snap -1 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:195: set_chunk oid 46 target oid 46 low_tier 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:195: set_chunk oid 46 offset: 1140224 length: 731136 target oid smithi10548823-46 low_tier offset: 1140224 snap -1 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:196: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:196: set_chunk oid 47 offset: 940032 length: 322560 target oid smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 940032 snap -1 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:197: set_chunk oid 48 target oid 48 low_tier 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:197: set_chunk oid 48 offset: 56320 length: 53248 target oid smithi10548823-48 low_tier offset: 56320 snap -1 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:198: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:198: set_chunk oid 49 offset: 1569280 length: 202240 target oid smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1569280 snap -1 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:199: set_chunk oid 50 target oid 50 low_tier 2024-04-20T02:50:42.684 INFO:tasks.rados.rados.0.smithi105.stdout:199: set_chunk oid 50 offset: 52736 length: 332800 target oid smithi10548823-50 low_tier offset: 52736 snap -1 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:200: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:200: set_chunk oid 1 offset: 1138688 length: 836096 target oid smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1138688 snap -1 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:201: set_chunk oid 2 target oid 2 low_tier 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:201: set_chunk oid 2 offset: 273920 length: 217088 target oid smithi10548823-2 low_tier offset: 273920 snap -1 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:202: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:202: set_chunk oid 3 offset: 1267200 length: 625664 target oid smithi10548823-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1267200 snap -1 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:203: set_chunk oid 4 target oid 4 low_tier 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:203: set_chunk oid 4 offset: 501248 length: 886784 target oid smithi10548823-4 low_tier offset: 501248 snap -1 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:188: finishing set_chunk to oid 39 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 398848 length: 704512 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:42.685 INFO:tasks.rados.rados.0.smithi105.stdout:188: done (15 left) 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:204: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:204: set_chunk oid 5 offset: 409088 length: 1093120 target oid smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 409088 snap -1 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:191: finishing set_chunk to oid 42 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 486400 length: 218112 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:191: done (15 left) 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:205: set_chunk oid 6 target oid 6 low_tier 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:205: set_chunk oid 6 offset: 565760 length: 426496 target oid smithi10548823-6 low_tier offset: 565760 snap -1 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:193: finishing set_chunk to oid 44 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 179712 length: 1415168 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:42.686 INFO:tasks.rados.rados.0.smithi105.stdout:194: finishing set_chunk to oid 45 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 447488 length: 1222656 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:195: finishing set_chunk to oid 46 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1140224 length: 731136 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:193: done (15 left) 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:194: done (14 left) 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:195: done (13 left) 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:206: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:206: set_chunk oid 7 offset: 337408 length: 551424 target oid smithi10548823-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 337408 snap -1 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:207: set_chunk oid 8 target oid 8 low_tier 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:207: set_chunk oid 8 offset: 1635840 length: 211968 target oid smithi10548823-8 low_tier offset: 1635840 snap -1 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:208: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.687 INFO:tasks.rados.rados.0.smithi105.stdout:208: set_chunk oid 9 offset: 267776 length: 442880 target oid smithi10548823-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 267776 snap -1 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:192: finishing set_chunk to oid 43 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 181248 length: 1238016 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:196: finishing set_chunk to oid 47 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 940032 length: 322560 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:198: finishing set_chunk to oid 49 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1569280 length: 202240 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:204: finishing set_chunk to oid 5 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 409088 length: 1093120 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:42.688 INFO:tasks.rados.rados.0.smithi105.stdout:192: done (15 left) 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:196: done (14 left) 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:198: done (13 left) 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:204: done (12 left) 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:209: set_chunk oid 10 target oid 10 low_tier 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:209: set_chunk oid 10 offset: 1585664 length: 376832 target oid smithi10548823-10 low_tier offset: 1585664 snap -1 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:210: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:210: set_chunk oid 11 offset: 140800 length: 39936 target oid smithi10548823-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 140800 snap -1 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:211: set_chunk oid 12 target oid 12 low_tier 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:211: set_chunk oid 12 offset: 301568 length: 107520 target oid smithi10548823-12 low_tier offset: 301568 snap -1 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:212: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:212: set_chunk oid 13 offset: 1200640 length: 645120 target oid smithi10548823-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1200640 snap -1 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:212: finishing set_chunk to oid 13 2024-04-20T02:50:42.689 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1200640 length: 645120 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:211: finishing set_chunk to oid 12 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 301568 length: 107520 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:211: done (15 left) 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:212: done (14 left) 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:213: set_chunk oid 14 target oid 14 low_tier 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:213: set_chunk oid 14 offset: 915456 length: 369152 target oid smithi10548823-14 low_tier offset: 915456 snap -1 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:214: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:214: set_chunk oid 15 offset: 132096 length: 1468416 target oid smithi10548823-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 132096 snap -1 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:214: finishing set_chunk to oid 15 2024-04-20T02:50:42.690 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 132096 length: 1468416 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:207: finishing set_chunk to oid 8 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1635840 length: 211968 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:207: done (15 left) 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:214: done (14 left) 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:215: set_chunk oid 16 target oid 16 low_tier 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:215: set_chunk oid 16 offset: 549888 length: 995328 target oid smithi10548823-16 low_tier offset: 549888 snap -1 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:216: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:216: set_chunk oid 17 offset: 103424 length: 34304 target oid smithi10548823-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 103424 snap -1 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:190: finishing set_chunk to oid 41 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1218048 length: 729600 2024-04-20T02:50:42.691 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:200: finishing set_chunk to oid 1 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1138688 length: 836096 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:205: finishing set_chunk to oid 6 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 565760 length: 426496 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:203: finishing set_chunk to oid 4 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 501248 length: 886784 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:206: finishing set_chunk to oid 7 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 337408 length: 551424 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:190: done (15 left) 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:200: done (14 left) 2024-04-20T02:50:42.692 INFO:tasks.rados.rados.0.smithi105.stdout:203: done (13 left) 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:205: done (12 left) 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:206: done (11 left) 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:217: set_chunk oid 18 target oid 18 low_tier 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:217: set_chunk oid 18 offset: 1579008 length: 262144 target oid smithi10548823-18 low_tier offset: 1579008 snap -1 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:218: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:218: set_chunk oid 19 offset: 281088 length: 1054208 target oid smithi10548823-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 281088 snap -1 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:219: set_chunk oid 20 target oid 20 low_tier 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:219: set_chunk oid 20 offset: 1034240 length: 214528 target oid smithi10548823-20 low_tier offset: 1034240 snap -1 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:220: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:220: set_chunk oid 21 offset: 62464 length: 377344 target oid smithi10548823-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 62464 snap -1 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:221: set_chunk oid 22 target oid 22 low_tier 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout:221: set_chunk oid 22 offset: 1272320 length: 826880 target oid smithi10548823-22 low_tier offset: 1272320 snap -1 2024-04-20T02:50:42.693 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:215: finishing set_chunk to oid 16 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 549888 length: 995328 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:213: finishing set_chunk to oid 14 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 915456 length: 369152 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:218: finishing set_chunk to oid 19 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 281088 length: 1054208 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:220: finishing set_chunk to oid 21 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 62464 length: 377344 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:42.694 INFO:tasks.rados.rados.0.smithi105.stdout:208: finishing set_chunk to oid 9 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 267776 length: 442880 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:189: finishing set_chunk to oid 40 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 21 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:201: finishing set_chunk to oid 2 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:202: finishing set_chunk to oid 3 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:197: finishing set_chunk to oid 48 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:210: finishing set_chunk to oid 11 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:216: finishing set_chunk to oid 17 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:42.695 INFO:tasks.rados.rados.0.smithi105.stdout:189: done (15 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:197: done (14 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:201: done (13 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:202: done (12 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:208: done (11 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:210: done (10 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:213: done (9 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:215: done (8 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:216: done (7 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:218: done (6 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:220: done (5 left) 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:222: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:222: set_chunk oid 23 offset: 1019392 length: 316416 target oid smithi10548823-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1019392 snap -1 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:223: set_chunk oid 24 target oid 24 low_tier 2024-04-20T02:50:42.696 INFO:tasks.rados.rados.0.smithi105.stdout:223: set_chunk oid 24 offset: 317952 length: 1897472 target oid smithi10548823-24 low_tier offset: 317952 snap -1 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:224: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:224: set_chunk oid 25 offset: 996864 length: 960000 target oid smithi10548823-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 996864 snap -1 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:225: set_chunk oid 26 target oid 26 low_tier 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:225: set_chunk oid 26 offset: 465408 length: 1903616 target oid smithi10548823-26 low_tier offset: 465408 snap -1 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:226: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:226: set_chunk oid 27 offset: 1188864 length: 336384 target oid smithi10548823-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1188864 snap -1 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:227: set_chunk oid 28 target oid 28 low_tier 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:227: set_chunk oid 28 offset: 28672 length: 835584 target oid smithi10548823-28 low_tier offset: 28672 snap -1 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:228: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:228: set_chunk oid 29 offset: 588288 length: 1777664 target oid smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 588288 snap -1 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:229: set_chunk oid 30 target oid 30 low_tier 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:229: set_chunk oid 30 offset: 1220096 length: 55808 target oid smithi10548823-30 low_tier offset: 1220096 snap -1 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:230: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.697 INFO:tasks.rados.rados.0.smithi105.stdout:230: set_chunk oid 31 offset: 1401344 length: 205824 target oid smithi10548823-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1401344 snap -1 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:231: set_chunk oid 32 target oid 32 low_tier 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:231: set_chunk oid 32 offset: 2129920 length: 195584 target oid smithi10548823-32 low_tier offset: 2129920 snap -1 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:232: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:232: set_chunk oid 33 offset: 547840 length: 231936 target oid smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 547840 snap -1 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:223: finishing set_chunk to oid 24 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 317952 length: 1897472 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:223: done (15 left) 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:233: set_chunk oid 34 target oid 34 low_tier 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:233: set_chunk oid 34 offset: 1840640 length: 707584 target oid smithi10548823-34 low_tier offset: 1840640 snap -1 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.698 INFO:tasks.rados.rados.0.smithi105.stdout:225: finishing set_chunk to oid 26 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 465408 length: 1903616 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:231: finishing set_chunk to oid 32 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 2129920 length: 195584 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:225: done (15 left) 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:231: done (14 left) 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:234: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:234: set_chunk oid 35 offset: 632832 length: 1702400 target oid smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 632832 snap -1 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:235: set_chunk oid 36 target oid 36 low_tier 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:235: set_chunk oid 36 offset: 179200 length: 747008 target oid smithi10548823-36 low_tier offset: 179200 snap -1 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.699 INFO:tasks.rados.rados.0.smithi105.stdout:224: finishing set_chunk to oid 25 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 996864 length: 960000 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:226: finishing set_chunk to oid 27 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1188864 length: 336384 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:228: finishing set_chunk to oid 29 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 588288 length: 1777664 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:229: finishing set_chunk to oid 30 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 1220096 length: 55808 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:230: finishing set_chunk to oid 31 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1401344 length: 205824 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:42.700 INFO:tasks.rados.rados.0.smithi105.stdout:232: finishing set_chunk to oid 33 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 547840 length: 231936 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:224: done (15 left) 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:226: done (14 left) 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:228: done (13 left) 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:229: done (12 left) 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:230: done (11 left) 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:232: done (10 left) 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:236: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:236: set_chunk oid 37 offset: 593920 length: 1899008 target oid smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 593920 snap -1 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:237: set_chunk oid 38 target oid 38 low_tier 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:237: set_chunk oid 38 offset: 397312 length: 1896960 target oid smithi10548823-38 low_tier offset: 397312 snap -1 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:238: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.701 INFO:tasks.rados.rados.0.smithi105.stdout:238: set_chunk oid 39 offset: 482304 length: 927744 target oid smithi10548823-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 482304 snap -1 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:239: set_chunk oid 40 target oid 40 low_tier 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:239: set_chunk oid 40 offset: 2023936 length: 491008 target oid smithi10548823-40 low_tier offset: 2023936 snap -1 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:240: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:240: set_chunk oid 41 offset: 1523712 length: 995328 target oid smithi10548823-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1523712 snap -1 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:241: set_chunk oid 42 target oid 42 low_tier 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:241: set_chunk oid 42 offset: 113664 length: 372224 target oid smithi10548823-42 low_tier offset: 113664 snap -1 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:235: finishing set_chunk to oid 36 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 179200 length: 747008 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:235: done (15 left) 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:242: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:242: set_chunk oid 43 offset: 365056 length: 957440 target oid smithi10548823-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 365056 snap -1 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.702 INFO:tasks.rados.rados.0.smithi105.stdout:234: finishing set_chunk to oid 35 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 632832 length: 1702400 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:234: done (15 left) 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:243: set_chunk oid 44 target oid 44 low_tier 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:243: set_chunk oid 44 offset: 316416 length: 2296320 target oid smithi10548823-44 low_tier offset: 316416 snap -1 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:238: finishing set_chunk to oid 39 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 482304 length: 927744 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:233: finishing set_chunk to oid 34 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1840640 length: 707584 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:233: done (15 left) 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:238: done (14 left) 2024-04-20T02:50:42.703 INFO:tasks.rados.rados.0.smithi105.stdout:244: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:244: set_chunk oid 45 offset: 432128 length: 178688 target oid smithi10548823-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 432128 snap -1 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:245: set_chunk oid 46 target oid 46 low_tier 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:245: set_chunk oid 46 offset: 890880 length: 1394688 target oid smithi10548823-46 low_tier offset: 890880 snap -1 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:236: finishing set_chunk to oid 37 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 593920 length: 1899008 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:236: done (15 left) 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:246: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:246: set_chunk oid 47 offset: 228864 length: 1000960 target oid smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 228864 snap -1 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:243: finishing set_chunk to oid 44 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 316416 length: 2296320 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:42.704 INFO:tasks.rados.rados.0.smithi105.stdout:241: finishing set_chunk to oid 42 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 113664 length: 372224 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:241: done (15 left) 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:243: done (14 left) 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:247: set_chunk oid 48 target oid 48 low_tier 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:247: set_chunk oid 48 offset: 152064 length: 1390592 target oid smithi10548823-48 low_tier offset: 152064 snap -1 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:248: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:248: set_chunk oid 49 offset: 2374144 length: 194048 target oid smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2374144 snap -1 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:237: finishing set_chunk to oid 38 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 397312 length: 1896960 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:240: finishing set_chunk to oid 41 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1523712 length: 995328 2024-04-20T02:50:42.705 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:237: done (15 left) 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:240: done (14 left) 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:248 retry set_chunk ! 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:245: finishing set_chunk to oid 46 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 890880 length: 1394688 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:245: done (13 left) 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 13 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:246: finishing set_chunk to oid 47 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 228864 length: 1000960 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:246: done (12 left) 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 12 2024-04-20T02:50:42.706 INFO:tasks.rados.rados.0.smithi105.stdout:219: finishing set_chunk to oid 20 2024-04-20T02:50:42.707 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:42.707 INFO:tasks.rados.rados.0.smithi105.stdout:221: finishing set_chunk to oid 22 2024-04-20T02:50:42.707 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:42.707 INFO:tasks.rados.rados.0.smithi105.stdout:219: done (11 left) 2024-04-20T02:50:42.707 INFO:tasks.rados.rados.0.smithi105.stdout:221: done (10 left) 2024-04-20T02:50:42.707 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 10 2024-04-20T02:50:42.708 INFO:tasks.rados.rados.0.smithi105.stdout:244: finishing set_chunk to oid 45 2024-04-20T02:50:42.708 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:42.708 INFO:tasks.rados.rados.0.smithi105.stdout:244: done (9 left) 2024-04-20T02:50:42.708 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 9 2024-04-20T02:50:42.708 INFO:tasks.rados.rados.0.smithi105.stdout:239: finishing set_chunk to oid 40 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 21 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout:239: done (8 left) 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 8 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout:242: finishing set_chunk to oid 43 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 15 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout:242: done (7 left) 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 7 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout:247: finishing set_chunk to oid 48 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout:247: done (6 left) 2024-04-20T02:50:42.709 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 6 2024-04-20T02:50:42.709 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:42.710 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:42.710 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:42.708+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:42.714 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:50:42.727 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:50:42.747 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:50:42.747 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:50:42.747 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:50:42.747 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:50:42.748 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:50:42.748 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:50:42.748 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-20T02:50:42.748 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:50:42.748 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:50:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:50:42.750 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:50:42.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:42.809+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:42.857 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:42.857 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:42.867 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:209: finishing set_chunk to oid 10 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:209: done (5 left) 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 5 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:199: finishing set_chunk to oid 50 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 6 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:199: done (4 left) 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 4 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:217: finishing set_chunk to oid 18 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:227: finishing set_chunk to oid 28 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:217: done (3 left) 2024-04-20T02:50:42.872 INFO:tasks.rados.rados.0.smithi105.stdout:227: done (2 left) 2024-04-20T02:50:42.873 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 2 2024-04-20T02:50:42.882 INFO:tasks.rados.rados.0.smithi105.stdout:222: finishing set_chunk to oid 23 2024-04-20T02:50:42.882 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:42.882 INFO:tasks.rados.rados.0.smithi105.stdout:222: done (1 left) 2024-04-20T02:50:42.882 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 1 2024-04-20T02:50:42.882 INFO:tasks.rados.rados.0.smithi105.stdout:248: finishing set_chunk to oid 49 2024-04-20T02:50:42.882 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:42.882 INFO:tasks.rados.rados.0.smithi105.stdout:248: done (0 left) 2024-04-20T02:50:42.882 INFO:tasks.rados.rados.0.smithi105.stdout:249: set_chunk oid 50 target oid 50 low_tier 2024-04-20T02:50:42.882 INFO:tasks.rados.rados.0.smithi105.stdout:249: set_chunk oid 50 offset: 2257920 length: 98304 target oid smithi10548823-50 low_tier offset: 2257920 snap -1 2024-04-20T02:50:42.882 INFO:tasks.rados.rados.0.smithi105.stdout:250: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:250: set_chunk oid 1 offset: 866304 length: 644608 target oid smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 866304 snap -1 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:251: set_chunk oid 2 target oid 2 low_tier 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:251: set_chunk oid 2 offset: 513024 length: 643584 target oid smithi10548823-2 low_tier offset: 513024 snap -1 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:252: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:252: set_chunk oid 3 offset: 1612800 length: 331264 target oid smithi10548823-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1612800 snap -1 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:253: set_chunk oid 4 target oid 4 low_tier 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:253: set_chunk oid 4 offset: 964096 length: 175616 target oid smithi10548823-4 low_tier offset: 964096 snap -1 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:254: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:254: set_chunk oid 5 offset: 109568 length: 1888768 target oid smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 109568 snap -1 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:255: set_chunk oid 6 target oid 6 low_tier 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:255: set_chunk oid 6 offset: 894976 length: 604160 target oid smithi10548823-6 low_tier offset: 894976 snap -1 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:256: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.883 INFO:tasks.rados.rados.0.smithi105.stdout:256: set_chunk oid 7 offset: 643584 length: 237056 target oid smithi10548823-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 643584 snap -1 2024-04-20T02:50:42.884 INFO:tasks.rados.rados.0.smithi105.stdout:257: set_chunk oid 8 target oid 8 low_tier 2024-04-20T02:50:42.884 INFO:tasks.rados.rados.0.smithi105.stdout:257: set_chunk oid 8 offset: 211968 length: 1364480 target oid smithi10548823-8 low_tier offset: 211968 snap -1 2024-04-20T02:50:42.884 INFO:tasks.rados.rados.0.smithi105.stdout:258: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.884 INFO:tasks.rados.rados.0.smithi105.stdout:258: set_chunk oid 9 offset: 824320 length: 664064 target oid smithi10548823-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 824320 snap -1 2024-04-20T02:50:42.884 INFO:tasks.rados.rados.0.smithi105.stdout:259: set_chunk oid 10 target oid 10 low_tier 2024-04-20T02:50:42.884 INFO:tasks.rados.rados.0.smithi105.stdout:259: set_chunk oid 10 offset: 1087488 length: 573440 target oid smithi10548823-10 low_tier offset: 1087488 snap -1 2024-04-20T02:50:42.884 INFO:tasks.rados.rados.0.smithi105.stdout:260: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.884 INFO:tasks.rados.rados.0.smithi105.stdout:260: set_chunk oid 11 offset: 217600 length: 1604096 target oid smithi10548823-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 217600 snap -1 2024-04-20T02:50:42.884 INFO:tasks.rados.rados.0.smithi105.stdout:261: set_chunk oid 12 target oid 12 low_tier 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:261: set_chunk oid 12 offset: 882688 length: 1176576 target oid smithi10548823-12 low_tier offset: 882688 snap -1 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:262: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:262: set_chunk oid 13 offset: 437248 length: 124928 target oid smithi10548823-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 437248 snap -1 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:263: set_chunk oid 14 target oid 14 low_tier 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:263: set_chunk oid 14 offset: 496128 length: 1471488 target oid smithi10548823-14 low_tier offset: 496128 snap -1 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:264: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:264: set_chunk oid 15 offset: 607744 length: 1445376 target oid smithi10548823-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 607744 snap -1 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:250: finishing set_chunk to oid 1 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 866304 length: 644608 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:250: done (15 left) 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:265: set_chunk oid 16 target oid 16 low_tier 2024-04-20T02:50:42.885 INFO:tasks.rados.rados.0.smithi105.stdout:265: set_chunk oid 16 offset: 571392 length: 1497600 target oid smithi10548823-16 low_tier offset: 571392 snap -1 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:253: finishing set_chunk to oid 4 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 964096 length: 175616 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:257: finishing set_chunk to oid 8 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 211968 length: 1364480 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:255: finishing set_chunk to oid 6 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 894976 length: 604160 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:260: finishing set_chunk to oid 11 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 217600 length: 1604096 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:42.886 INFO:tasks.rados.rados.0.smithi105.stdout:253: done (15 left) 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:255: done (14 left) 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:257: done (13 left) 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:260: done (12 left) 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:266: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:266: set_chunk oid 17 offset: 117248 length: 1511424 target oid smithi10548823-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 117248 snap -1 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:267: set_chunk oid 18 target oid 18 low_tier 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:267: set_chunk oid 18 offset: 1963520 length: 171520 target oid smithi10548823-18 low_tier offset: 1963520 snap -1 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:268: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:268: set_chunk oid 19 offset: 442880 length: 1018880 target oid smithi10548823-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 442880 snap -1 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:269: set_chunk oid 20 target oid 20 low_tier 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:269: set_chunk oid 20 offset: 376320 length: 1071104 target oid smithi10548823-20 low_tier offset: 376320 snap -1 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:256: finishing set_chunk to oid 7 2024-04-20T02:50:42.887 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 643584 length: 237056 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:261: finishing set_chunk to oid 12 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 882688 length: 1176576 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:251: finishing set_chunk to oid 2 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 513024 length: 643584 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:252: finishing set_chunk to oid 3 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1612800 length: 331264 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:254: finishing set_chunk to oid 5 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 109568 length: 1888768 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:259: finishing set_chunk to oid 10 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 1087488 length: 573440 2024-04-20T02:50:42.888 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:264: finishing set_chunk to oid 15 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 607744 length: 1445376 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:263: finishing set_chunk to oid 14 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 496128 length: 1471488 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:258: finishing set_chunk to oid 9 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 824320 length: 664064 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:251: done (15 left) 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:252: done (14 left) 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:254: done (13 left) 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:256: done (12 left) 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:258: done (11 left) 2024-04-20T02:50:42.889 INFO:tasks.rados.rados.0.smithi105.stdout:259: done (10 left) 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:261: done (9 left) 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:263: done (8 left) 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:264: done (7 left) 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:270: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:270: set_chunk oid 21 offset: 1128960 length: 908288 target oid smithi10548823-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1128960 snap -1 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:271: set_chunk oid 22 target oid 22 low_tier 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:271: set_chunk oid 22 offset: 783360 length: 566784 target oid smithi10548823-22 low_tier offset: 783360 snap -1 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:272: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:272: set_chunk oid 23 offset: 1619968 length: 185856 target oid smithi10548823-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1619968 snap -1 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:273: set_chunk oid 24 target oid 24 low_tier 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:273: set_chunk oid 24 offset: 774656 length: 11264 target oid smithi10548823-24 low_tier offset: 774656 snap -1 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:274: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:274: set_chunk oid 25 offset: 1229312 length: 208896 target oid smithi10548823-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1229312 snap -1 2024-04-20T02:50:42.890 INFO:tasks.rados.rados.0.smithi105.stdout:275: set_chunk oid 26 target oid 26 low_tier 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:275: set_chunk oid 26 offset: 835072 length: 1406464 target oid smithi10548823-26 low_tier offset: 835072 snap -1 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:276: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:276: set_chunk oid 27 offset: 208896 length: 629760 target oid smithi10548823-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 208896 snap -1 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:277: set_chunk oid 28 target oid 28 low_tier 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:277: set_chunk oid 28 offset: 341504 length: 1717248 target oid smithi10548823-28 low_tier offset: 341504 snap -1 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:278: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:278: set_chunk oid 29 offset: 241152 length: 451072 target oid smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 241152 snap -1 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:265: finishing set_chunk to oid 16 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 571392 length: 1497600 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:265: done (15 left) 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:279: set_chunk oid 30 target oid 30 low_tier 2024-04-20T02:50:42.891 INFO:tasks.rados.rados.0.smithi105.stdout:279: set_chunk oid 30 offset: 757248 length: 1101824 target oid smithi10548823-30 low_tier offset: 757248 snap -1 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:266: finishing set_chunk to oid 17 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 117248 length: 1511424 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:268: finishing set_chunk to oid 19 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 442880 length: 1018880 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:266: done (15 left) 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:268: done (14 left) 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:280: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:280: set_chunk oid 31 offset: 824320 length: 416768 target oid smithi10548823-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 824320 snap -1 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:281: set_chunk oid 32 target oid 32 low_tier 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:281: set_chunk oid 32 offset: 1472512 length: 1047552 target oid smithi10548823-32 low_tier offset: 1472512 snap -1 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:275: finishing set_chunk to oid 26 2024-04-20T02:50:42.892 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 835072 length: 1406464 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:277: finishing set_chunk to oid 28 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 341504 length: 1717248 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:270: finishing set_chunk to oid 21 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1128960 length: 908288 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:278: finishing set_chunk to oid 29 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 241152 length: 451072 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:279: finishing set_chunk to oid 30 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 757248 length: 1101824 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:280: finishing set_chunk to oid 31 2024-04-20T02:50:42.893 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 824320 length: 416768 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:281: finishing set_chunk to oid 32 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1472512 length: 1047552 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:269: finishing set_chunk to oid 20 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 376320 length: 1071104 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:271: finishing set_chunk to oid 22 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 783360 length: 566784 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:274: finishing set_chunk to oid 25 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1229312 length: 208896 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:269: done (15 left) 2024-04-20T02:50:42.894 INFO:tasks.rados.rados.0.smithi105.stdout:270: done (14 left) 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:271: done (13 left) 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:274: done (12 left) 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:275: done (11 left) 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:277: done (10 left) 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:278: done (9 left) 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:279: done (8 left) 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:280: done (7 left) 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:281: done (6 left) 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:282: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:282: set_chunk oid 33 offset: 857600 length: 163840 target oid smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 857600 snap -1 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:283: set_chunk oid 34 target oid 34 low_tier 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:283: set_chunk oid 34 offset: 580608 length: 762368 target oid smithi10548823-34 low_tier offset: 580608 snap -1 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:284: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.895 INFO:tasks.rados.rados.0.smithi105.stdout:284: set_chunk oid 35 offset: 970240 length: 191488 target oid smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 970240 snap -1 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:285: set_chunk oid 36 target oid 36 low_tier 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:285: set_chunk oid 36 offset: 1751552 length: 809472 target oid smithi10548823-36 low_tier offset: 1751552 snap -1 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:286: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:286: set_chunk oid 37 offset: 676352 length: 1728512 target oid smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 676352 snap -1 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:287: set_chunk oid 38 target oid 38 low_tier 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:287: set_chunk oid 38 offset: 188416 length: 1501184 target oid smithi10548823-38 low_tier offset: 188416 snap -1 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:288: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:288: set_chunk oid 39 offset: 1421824 length: 548352 target oid smithi10548823-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1421824 snap -1 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:289: set_chunk oid 40 target oid 40 low_tier 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:289: set_chunk oid 40 offset: 1727488 length: 242688 target oid smithi10548823-40 low_tier offset: 1727488 snap -1 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:290: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:290: set_chunk oid 41 offset: 1048576 length: 1314816 target oid smithi10548823-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1048576 snap -1 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:291: set_chunk oid 42 target oid 42 low_tier 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:291: set_chunk oid 42 offset: 165376 length: 2112000 target oid smithi10548823-42 low_tier offset: 165376 snap -1 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.896 INFO:tasks.rados.rados.0.smithi105.stdout:276: finishing set_chunk to oid 27 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 208896 length: 629760 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout:276: done (15 left) 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout:292: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout:292: set_chunk oid 43 offset: 1507840 length: 946688 target oid smithi10548823-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1507840 snap -1 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout:283: finishing set_chunk to oid 34 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 580608 length: 762368 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout:290: finishing set_chunk to oid 41 2024-04-20T02:50:42.897 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1048576 length: 1314816 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:282: finishing set_chunk to oid 33 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 857600 length: 163840 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:289: finishing set_chunk to oid 40 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1727488 length: 242688 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:286: finishing set_chunk to oid 37 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 676352 length: 1728512 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:291: finishing set_chunk to oid 42 2024-04-20T02:50:42.898 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 165376 length: 2112000 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:282: done (15 left) 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:283: done (14 left) 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:286: done (13 left) 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:289: done (12 left) 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:290: done (11 left) 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:291: done (10 left) 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:293: set_chunk oid 44 target oid 44 low_tier 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:293: set_chunk oid 44 offset: 1764864 length: 109056 target oid smithi10548823-44 low_tier offset: 1764864 snap -1 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:294: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:294: set_chunk oid 45 offset: 876032 length: 1580032 target oid smithi10548823-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 876032 snap -1 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:295: set_chunk oid 46 target oid 46 low_tier 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:295: set_chunk oid 46 offset: 1584128 length: 51712 target oid smithi10548823-46 low_tier offset: 1584128 snap -1 2024-04-20T02:50:42.899 INFO:tasks.rados.rados.0.smithi105.stdout:296: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:296: set_chunk oid 47 offset: 1231872 length: 1447936 target oid smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1231872 snap -1 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:297: set_chunk oid 48 target oid 48 low_tier 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:297: set_chunk oid 48 offset: 89600 length: 694272 target oid smithi10548823-48 low_tier offset: 89600 snap -1 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:298: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:298: set_chunk oid 49 offset: 2108416 length: 667648 target oid smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2108416 snap -1 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:287: finishing set_chunk to oid 38 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 188416 length: 1501184 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:288: finishing set_chunk to oid 39 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1421824 length: 548352 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:42.900 INFO:tasks.rados.rados.0.smithi105.stdout:292: finishing set_chunk to oid 43 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1507840 length: 946688 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout:287: done (15 left) 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout:288: done (14 left) 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout:292: done (13 left) 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout:298 retry set_chunk ! 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 13 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout:294: finishing set_chunk to oid 45 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 876032 length: 1580032 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout:294: done (12 left) 2024-04-20T02:50:42.901 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 12 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:296: finishing set_chunk to oid 47 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1231872 length: 1447936 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:296: done (11 left) 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 11 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:298: finishing set_chunk to oid 49 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2108416 length: 667648 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:298: done (10 left) 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 10 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:297: finishing set_chunk to oid 48 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 89600 length: 694272 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:42.902 INFO:tasks.rados.rados.0.smithi105.stdout:297: done (9 left) 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 9 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout:249: finishing set_chunk to oid 50 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 6 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout:249: done (8 left) 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 8 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout:272: finishing set_chunk to oid 23 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout:272: done (7 left) 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 7 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout:262: finishing set_chunk to oid 13 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout:262: done (6 left) 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 6 2024-04-20T02:50:42.903 INFO:tasks.rados.rados.0.smithi105.stdout:267: finishing set_chunk to oid 18 2024-04-20T02:50:42.904 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:42.904 INFO:tasks.rados.rados.0.smithi105.stdout:267: done (5 left) 2024-04-20T02:50:42.904 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 5 2024-04-20T02:50:42.904 INFO:tasks.rados.rados.0.smithi105.stdout:273: finishing set_chunk to oid 24 2024-04-20T02:50:42.904 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:42.904 INFO:tasks.rados.rados.0.smithi105.stdout:273: done (4 left) 2024-04-20T02:50:42.904 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 4 2024-04-20T02:50:42.920 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:42.913+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout:284: finishing set_chunk to oid 35 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 3 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout:284: done (3 left) 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 3 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout:285: finishing set_chunk to oid 36 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 18 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout:285: done (2 left) 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 2 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout:293: finishing set_chunk to oid 44 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout:293: done (1 left) 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 1 2024-04-20T02:50:42.966 INFO:tasks.rados.rados.0.smithi105.stdout:295: finishing set_chunk to oid 46 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:295: done (0 left) 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:299: set_chunk oid 50 target oid 50 low_tier 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:299: set_chunk oid 50 offset: 1026560 length: 911872 target oid smithi10548823-50 low_tier offset: 1026560 snap -1 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:300: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:300: set_chunk oid 1 offset: 568832 length: 1029120 target oid smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 568832 snap -1 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:301: set_chunk oid 2 target oid 2 low_tier 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:301: set_chunk oid 2 offset: 1561088 length: 25600 target oid smithi10548823-2 low_tier offset: 1561088 snap -1 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:302: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:302: set_chunk oid 3 offset: 464384 length: 514560 target oid smithi10548823-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 464384 snap -1 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:303: set_chunk oid 4 target oid 4 low_tier 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:303: set_chunk oid 4 offset: 1035264 length: 202240 target oid smithi10548823-4 low_tier offset: 1035264 snap -1 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:304: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.967 INFO:tasks.rados.rados.0.smithi105.stdout:304: set_chunk oid 5 offset: 1256448 length: 389632 target oid smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1256448 snap -1 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:305: set_chunk oid 6 target oid 6 low_tier 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:305: set_chunk oid 6 offset: 1133056 length: 621568 target oid smithi10548823-6 low_tier offset: 1133056 snap -1 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:306: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:306: set_chunk oid 7 offset: 275456 length: 242688 target oid smithi10548823-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 275456 snap -1 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:307: set_chunk oid 8 target oid 8 low_tier 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:307: set_chunk oid 8 offset: 367104 length: 1670656 target oid smithi10548823-8 low_tier offset: 367104 snap -1 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:308: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:308: set_chunk oid 9 offset: 114176 length: 61952 target oid smithi10548823-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 114176 snap -1 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:309: set_chunk oid 10 target oid 10 low_tier 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:309: set_chunk oid 10 offset: 1053696 length: 941056 target oid smithi10548823-10 low_tier offset: 1053696 snap -1 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:310: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:310: set_chunk oid 11 offset: 203776 length: 1729536 target oid smithi10548823-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 203776 snap -1 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:311: set_chunk oid 12 target oid 12 low_tier 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:311: set_chunk oid 12 offset: 1199616 length: 804864 target oid smithi10548823-12 low_tier offset: 1199616 snap -1 2024-04-20T02:50:42.968 INFO:tasks.rados.rados.0.smithi105.stdout:312: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:312: set_chunk oid 13 offset: 399872 length: 1462784 target oid smithi10548823-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 399872 snap -1 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:313: set_chunk oid 14 target oid 14 low_tier 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:313: set_chunk oid 14 offset: 142336 length: 1240576 target oid smithi10548823-14 low_tier offset: 142336 snap -1 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:314: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:314: set_chunk oid 15 offset: 316416 length: 605184 target oid smithi10548823-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 316416 snap -1 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:301: finishing set_chunk to oid 2 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1561088 length: 25600 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:302: finishing set_chunk to oid 3 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 464384 length: 514560 2024-04-20T02:50:42.969 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:300: finishing set_chunk to oid 1 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 568832 length: 1029120 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:300: done (15 left) 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:301: done (14 left) 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:302: done (13 left) 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:315: set_chunk oid 16 target oid 16 low_tier 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:315: set_chunk oid 16 offset: 564736 length: 1550848 target oid smithi10548823-16 low_tier offset: 564736 snap -1 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:316: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:316: set_chunk oid 17 offset: 409088 length: 275968 target oid smithi10548823-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 409088 snap -1 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:317: set_chunk oid 18 target oid 18 low_tier 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout:317: set_chunk oid 18 offset: 498688 length: 838144 target oid smithi10548823-18 low_tier offset: 498688 snap -1 2024-04-20T02:50:42.970 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:310: finishing set_chunk to oid 11 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 203776 length: 1729536 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:310: done (15 left) 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:318: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:318: set_chunk oid 19 offset: 1013760 length: 1041408 target oid smithi10548823-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1013760 snap -1 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:307: finishing set_chunk to oid 8 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 367104 length: 1670656 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:307: done (15 left) 2024-04-20T02:50:42.971 INFO:tasks.rados.rados.0.smithi105.stdout:319: set_chunk oid 20 target oid 20 low_tier 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:319: set_chunk oid 20 offset: 295424 length: 139776 target oid smithi10548823-20 low_tier offset: 295424 snap -1 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:313: finishing set_chunk to oid 14 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 142336 length: 1240576 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:311: finishing set_chunk to oid 12 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1199616 length: 804864 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:303: finishing set_chunk to oid 4 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1035264 length: 202240 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:305: finishing set_chunk to oid 6 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1133056 length: 621568 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:42.972 INFO:tasks.rados.rados.0.smithi105.stdout:306: finishing set_chunk to oid 7 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 275456 length: 242688 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:303: done (15 left) 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:305: done (14 left) 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:306: done (13 left) 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:311: done (12 left) 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:313: done (11 left) 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:320: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:320: set_chunk oid 21 offset: 910336 length: 70656 target oid smithi10548823-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 910336 snap -1 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:321: set_chunk oid 22 target oid 22 low_tier 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:321: set_chunk oid 22 offset: 1235968 length: 216064 target oid smithi10548823-22 low_tier offset: 1235968 snap -1 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:322: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:322: set_chunk oid 23 offset: 983552 length: 789504 target oid smithi10548823-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 983552 snap -1 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:323: set_chunk oid 24 target oid 24 low_tier 2024-04-20T02:50:42.973 INFO:tasks.rados.rados.0.smithi105.stdout:323: set_chunk oid 24 offset: 788992 length: 504320 target oid smithi10548823-24 low_tier offset: 788992 snap -1 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:324: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:324: set_chunk oid 25 offset: 1991168 length: 175104 target oid smithi10548823-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1991168 snap -1 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:312: finishing set_chunk to oid 13 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 399872 length: 1462784 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:309: finishing set_chunk to oid 10 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 1053696 length: 941056 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:309: done (15 left) 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:312: done (14 left) 2024-04-20T02:50:42.974 INFO:tasks.rados.rados.0.smithi105.stdout:325: set_chunk oid 26 target oid 26 low_tier 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout:325: set_chunk oid 26 offset: 1523200 length: 730624 target oid smithi10548823-26 low_tier offset: 1523200 snap -1 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout:326: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout:326: set_chunk oid 27 offset: 1796608 length: 219648 target oid smithi10548823-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1796608 snap -1 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout:314: finishing set_chunk to oid 15 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 316416 length: 605184 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout:315: finishing set_chunk to oid 16 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 564736 length: 1550848 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:42.975 INFO:tasks.rados.rados.0.smithi105.stdout:317: finishing set_chunk to oid 18 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 498688 length: 838144 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:318: finishing set_chunk to oid 19 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1013760 length: 1041408 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:314: done (15 left) 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:315: done (14 left) 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:317: done (13 left) 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:318: done (12 left) 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:327: set_chunk oid 28 target oid 28 low_tier 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:327: set_chunk oid 28 offset: 1059840 length: 6144 target oid smithi10548823-28 low_tier offset: 1059840 snap -1 2024-04-20T02:50:42.976 INFO:tasks.rados.rados.0.smithi105.stdout:328: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:328: set_chunk oid 29 offset: 933888 length: 749056 target oid smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 933888 snap -1 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:329: set_chunk oid 30 target oid 30 low_tier 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:329: set_chunk oid 30 offset: 1228800 length: 1070080 target oid smithi10548823-30 low_tier offset: 1228800 snap -1 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:330: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:330: set_chunk oid 31 offset: 549888 length: 95232 target oid smithi10548823-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 549888 snap -1 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:323: finishing set_chunk to oid 24 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 788992 length: 504320 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:323: done (15 left) 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:331: set_chunk oid 32 target oid 32 low_tier 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:331: set_chunk oid 32 offset: 600064 length: 1565696 target oid smithi10548823-32 low_tier offset: 600064 snap -1 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.977 INFO:tasks.rados.rados.0.smithi105.stdout:325: finishing set_chunk to oid 26 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1523200 length: 730624 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:325: done (15 left) 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:332: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:332: set_chunk oid 33 offset: 573440 length: 964608 target oid smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 573440 snap -1 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:327: finishing set_chunk to oid 28 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1059840 length: 6144 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:327: done (15 left) 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:333: set_chunk oid 34 target oid 34 low_tier 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:333: set_chunk oid 34 offset: 1448448 length: 1070080 target oid smithi10548823-34 low_tier offset: 1448448 snap -1 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:329: finishing set_chunk to oid 30 2024-04-20T02:50:42.978 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 1228800 length: 1070080 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:328: finishing set_chunk to oid 29 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 933888 length: 749056 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:331: finishing set_chunk to oid 32 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 600064 length: 1565696 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:332: finishing set_chunk to oid 33 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 573440 length: 964608 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:328: done (15 left) 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:329: done (14 left) 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:331: done (13 left) 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:332: done (12 left) 2024-04-20T02:50:42.979 INFO:tasks.rados.rados.0.smithi105.stdout:334: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:334: set_chunk oid 35 offset: 1823744 length: 595456 target oid smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1823744 snap -1 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:335: set_chunk oid 36 target oid 36 low_tier 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:335: set_chunk oid 36 offset: 1799680 length: 699392 target oid smithi10548823-36 low_tier offset: 1799680 snap -1 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:336: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:336: set_chunk oid 37 offset: 311296 length: 924672 target oid smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 311296 snap -1 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:337: set_chunk oid 38 target oid 38 low_tier 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:337: set_chunk oid 38 offset: 1648128 length: 276992 target oid smithi10548823-38 low_tier offset: 1648128 snap -1 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:319: finishing set_chunk to oid 20 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 295424 length: 139776 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:324: finishing set_chunk to oid 25 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1991168 length: 175104 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:42.980 INFO:tasks.rados.rados.0.smithi105.stdout:321: finishing set_chunk to oid 22 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1235968 length: 216064 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:322: finishing set_chunk to oid 23 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 983552 length: 789504 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:319: done (15 left) 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:321: done (14 left) 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:322: done (13 left) 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:324: done (12 left) 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:338: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:338: set_chunk oid 39 offset: 802816 length: 1446400 target oid smithi10548823-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 802816 snap -1 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:339: set_chunk oid 40 target oid 40 low_tier 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:339: set_chunk oid 40 offset: 791040 length: 856064 target oid smithi10548823-40 low_tier offset: 791040 snap -1 2024-04-20T02:50:42.981 INFO:tasks.rados.rados.0.smithi105.stdout:340: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:340: set_chunk oid 41 offset: 1189888 length: 1048064 target oid smithi10548823-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1189888 snap -1 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:341: set_chunk oid 42 target oid 42 low_tier 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:341: set_chunk oid 42 offset: 2089472 length: 553472 target oid smithi10548823-42 low_tier offset: 2089472 snap -1 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:335: finishing set_chunk to oid 36 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1799680 length: 699392 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:333: finishing set_chunk to oid 34 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1448448 length: 1070080 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:333: done (15 left) 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:335: done (14 left) 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:342: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:342: set_chunk oid 43 offset: 219136 length: 102912 target oid smithi10548823-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 219136 snap -1 2024-04-20T02:50:42.982 INFO:tasks.rados.rados.0.smithi105.stdout:343: set_chunk oid 44 target oid 44 low_tier 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:343: set_chunk oid 44 offset: 997888 length: 47104 target oid smithi10548823-44 low_tier offset: 997888 snap -1 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:334: finishing set_chunk to oid 35 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1823744 length: 595456 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:336: finishing set_chunk to oid 37 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 311296 length: 924672 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:337: finishing set_chunk to oid 38 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 1648128 length: 276992 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:334: done (15 left) 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:336: done (14 left) 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:337: done (13 left) 2024-04-20T02:50:42.983 INFO:tasks.rados.rados.0.smithi105.stdout:344: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:344: set_chunk oid 45 offset: 594944 length: 1798144 target oid smithi10548823-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 594944 snap -1 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:345: set_chunk oid 46 target oid 46 low_tier 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:345: set_chunk oid 46 offset: 977408 length: 1413120 target oid smithi10548823-46 low_tier offset: 977408 snap -1 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:346: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:346: set_chunk oid 47 offset: 255488 length: 1593856 target oid smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 255488 snap -1 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:339: finishing set_chunk to oid 40 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 791040 length: 856064 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:338: finishing set_chunk to oid 39 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 802816 length: 1446400 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:42.984 INFO:tasks.rados.rados.0.smithi105.stdout:340: finishing set_chunk to oid 41 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1189888 length: 1048064 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:341: finishing set_chunk to oid 42 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 2089472 length: 553472 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:346: finishing set_chunk to oid 47 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 255488 length: 1593856 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:338: done (15 left) 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:339: done (14 left) 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:340: done (13 left) 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:341: done (12 left) 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:346: done (11 left) 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:347: set_chunk oid 48 target oid 48 low_tier 2024-04-20T02:50:42.985 INFO:tasks.rados.rados.0.smithi105.stdout:347: set_chunk oid 48 offset: 308224 length: 2039296 target oid smithi10548823-48 low_tier offset: 308224 snap -1 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:348: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:348: set_chunk oid 49 offset: 81408 length: 647680 target oid smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 81408 snap -1 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:348 retry set_chunk ! 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 13 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:343: finishing set_chunk to oid 44 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 997888 length: 47104 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:344: finishing set_chunk to oid 45 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 594944 length: 1798144 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:345: finishing set_chunk to oid 46 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 977408 length: 1413120 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:343: done (12 left) 2024-04-20T02:50:42.986 INFO:tasks.rados.rados.0.smithi105.stdout:344: done (11 left) 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout:345: done (10 left) 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 10 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout:347: finishing set_chunk to oid 48 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 308224 length: 2039296 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout:347: done (9 left) 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 9 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout:299: finishing set_chunk to oid 50 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 6 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout:299: done (8 left) 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 8 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout:320: finishing set_chunk to oid 21 2024-04-20T02:50:42.987 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout:320: done (7 left) 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 7 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout:330: finishing set_chunk to oid 31 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout:330: done (6 left) 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 6 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout:326: finishing set_chunk to oid 27 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout:304: finishing set_chunk to oid 5 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout:304: done (5 left) 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout:326: done (4 left) 2024-04-20T02:50:42.988 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 4 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout:348: finishing set_chunk to oid 49 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout:348: done (3 left) 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 3 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout:342: finishing set_chunk to oid 43 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 15 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout:316: finishing set_chunk to oid 17 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout:316: done (2 left) 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout:342: done (1 left) 2024-04-20T02:50:42.989 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 1 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:308: finishing set_chunk to oid 9 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 12 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:308: done (0 left) 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:349: set_chunk oid 50 target oid 50 low_tier 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:349: set_chunk oid 50 offset: 333312 length: 1810944 target oid smithi10548823-50 low_tier offset: 333312 snap -1 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:350: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:350: set_chunk oid 1 offset: 795136 length: 949248 target oid smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 795136 snap -1 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:351: set_chunk oid 2 target oid 2 low_tier 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:351: set_chunk oid 2 offset: 1639936 length: 9216 target oid smithi10548823-2 low_tier offset: 1639936 snap -1 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:352: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:352: set_chunk oid 3 offset: 885248 length: 428032 target oid smithi10548823-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 885248 snap -1 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:353: set_chunk oid 4 target oid 4 low_tier 2024-04-20T02:50:42.990 INFO:tasks.rados.rados.0.smithi105.stdout:353: set_chunk oid 4 offset: 29696 length: 1314816 target oid smithi10548823-4 low_tier offset: 29696 snap -1 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:354: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:354: set_chunk oid 5 offset: 1371136 length: 178688 target oid smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1371136 snap -1 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:355: set_chunk oid 6 target oid 6 low_tier 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:355: set_chunk oid 6 offset: 1085440 length: 103424 target oid smithi10548823-6 low_tier offset: 1085440 snap -1 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:356: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:356: set_chunk oid 7 offset: 94720 length: 1923584 target oid smithi10548823-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 94720 snap -1 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:357: set_chunk oid 8 target oid 8 low_tier 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:357: set_chunk oid 8 offset: 429568 length: 608768 target oid smithi10548823-8 low_tier offset: 429568 snap -1 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:358: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:358: set_chunk oid 9 offset: 258048 length: 1323520 target oid smithi10548823-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 258048 snap -1 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:359: set_chunk oid 10 target oid 10 low_tier 2024-04-20T02:50:42.991 INFO:tasks.rados.rados.0.smithi105.stdout:359: set_chunk oid 10 offset: 1463296 length: 438272 target oid smithi10548823-10 low_tier offset: 1463296 snap -1 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:360: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:360: set_chunk oid 11 offset: 228864 length: 974848 target oid smithi10548823-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 228864 snap -1 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:361: set_chunk oid 12 target oid 12 low_tier 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:361: set_chunk oid 12 offset: 268800 length: 1836032 target oid smithi10548823-12 low_tier offset: 268800 snap -1 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:362: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:362: set_chunk oid 13 offset: 663552 length: 83968 target oid smithi10548823-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 663552 snap -1 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:363: set_chunk oid 14 target oid 14 low_tier 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:363: set_chunk oid 14 offset: 430080 length: 310784 target oid smithi10548823-14 low_tier offset: 430080 snap -1 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:364: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:364: set_chunk oid 15 offset: 785920 length: 238592 target oid smithi10548823-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 785920 snap -1 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:351: finishing set_chunk to oid 2 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1639936 length: 9216 2024-04-20T02:50:42.992 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:349: finishing set_chunk to oid 50 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 333312 length: 1810944 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:352: finishing set_chunk to oid 3 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 885248 length: 428032 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:349: done (15 left) 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:351: done (14 left) 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:352: done (13 left) 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:365: set_chunk oid 16 target oid 16 low_tier 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:365: set_chunk oid 16 offset: 254976 length: 717312 target oid smithi10548823-16 low_tier offset: 254976 snap -1 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:366: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:366: set_chunk oid 17 offset: 1578496 length: 336384 target oid smithi10548823-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1578496 snap -1 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:367: set_chunk oid 18 target oid 18 low_tier 2024-04-20T02:50:42.993 INFO:tasks.rados.rados.0.smithi105.stdout:367: set_chunk oid 18 offset: 461312 length: 1129472 target oid smithi10548823-18 low_tier offset: 461312 snap -1 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:357: finishing set_chunk to oid 8 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 429568 length: 608768 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:354: finishing set_chunk to oid 5 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1371136 length: 178688 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:359: finishing set_chunk to oid 10 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 1463296 length: 438272 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:354: done (15 left) 2024-04-20T02:50:42.994 INFO:tasks.rados.rados.0.smithi105.stdout:357: done (14 left) 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:359: done (13 left) 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:368: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:368: set_chunk oid 19 offset: 344576 length: 334848 target oid smithi10548823-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 344576 snap -1 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:369: set_chunk oid 20 target oid 20 low_tier 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:369: set_chunk oid 20 offset: 1244160 length: 815104 target oid smithi10548823-20 low_tier offset: 1244160 snap -1 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:370: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:370: set_chunk oid 21 offset: 734208 length: 447488 target oid smithi10548823-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 734208 snap -1 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:350: finishing set_chunk to oid 1 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 795136 length: 949248 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:355: finishing set_chunk to oid 6 2024-04-20T02:50:42.995 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1085440 length: 103424 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:364: finishing set_chunk to oid 15 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 785920 length: 238592 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:360: finishing set_chunk to oid 11 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 228864 length: 974848 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:353: finishing set_chunk to oid 4 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 29696 length: 1314816 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:365: finishing set_chunk to oid 16 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 254976 length: 717312 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:367: finishing set_chunk to oid 18 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 461312 length: 1129472 2024-04-20T02:50:42.996 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:363: finishing set_chunk to oid 14 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 430080 length: 310784 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:368: finishing set_chunk to oid 19 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 344576 length: 334848 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:366: finishing set_chunk to oid 17 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1578496 length: 336384 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:370: finishing set_chunk to oid 21 2024-04-20T02:50:42.997 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 734208 length: 447488 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:356: finishing set_chunk to oid 7 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 94720 length: 1923584 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:369: finishing set_chunk to oid 20 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1244160 length: 815104 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:361: finishing set_chunk to oid 12 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 268800 length: 1836032 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:350: done (15 left) 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:353: done (14 left) 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:355: done (13 left) 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:356: done (12 left) 2024-04-20T02:50:42.998 INFO:tasks.rados.rados.0.smithi105.stdout:360: done (11 left) 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:361: done (10 left) 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:363: done (9 left) 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:364: done (8 left) 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:365: done (7 left) 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:366: done (6 left) 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:367: done (5 left) 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:368: done (4 left) 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:369: done (3 left) 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:370: done (2 left) 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:371: set_chunk oid 22 target oid 22 low_tier 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:371: set_chunk oid 22 offset: 851968 length: 203776 target oid smithi10548823-22 low_tier offset: 851968 snap -1 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:372: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:372: set_chunk oid 23 offset: 812544 length: 267264 target oid smithi10548823-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 812544 snap -1 2024-04-20T02:50:42.999 INFO:tasks.rados.rados.0.smithi105.stdout:373: set_chunk oid 24 target oid 24 low_tier 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:373: set_chunk oid 24 offset: 1049600 length: 536576 target oid smithi10548823-24 low_tier offset: 1049600 snap -1 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:374: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:374: set_chunk oid 25 offset: 1440256 length: 412160 target oid smithi10548823-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1440256 snap -1 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:375: set_chunk oid 26 target oid 26 low_tier 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:375: set_chunk oid 26 offset: 451584 length: 956416 target oid smithi10548823-26 low_tier offset: 451584 snap -1 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:376: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:376: set_chunk oid 27 offset: 377344 length: 936448 target oid smithi10548823-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 377344 snap -1 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:377: set_chunk oid 28 target oid 28 low_tier 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:377: set_chunk oid 28 offset: 756736 length: 618496 target oid smithi10548823-28 low_tier offset: 756736 snap -1 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:378: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:378: set_chunk oid 29 offset: 674816 length: 1573888 target oid smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 674816 snap -1 2024-04-20T02:50:43.000 INFO:tasks.rados.rados.0.smithi105.stdout:379: set_chunk oid 30 target oid 30 low_tier 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:379: set_chunk oid 30 offset: 63488 length: 2009088 target oid smithi10548823-30 low_tier offset: 63488 snap -1 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:380: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:380: set_chunk oid 31 offset: 1459200 length: 518656 target oid smithi10548823-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1459200 snap -1 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:381: set_chunk oid 32 target oid 32 low_tier 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:381: set_chunk oid 32 offset: 449024 length: 1671680 target oid smithi10548823-32 low_tier offset: 449024 snap -1 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:382: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:382: set_chunk oid 33 offset: 1776640 length: 70656 target oid smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1776640 snap -1 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:383: set_chunk oid 34 target oid 34 low_tier 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:383: set_chunk oid 34 offset: 220672 length: 1436672 target oid smithi10548823-34 low_tier offset: 220672 snap -1 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:384: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout:384: set_chunk oid 35 offset: 1246720 length: 767488 target oid smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1246720 snap -1 2024-04-20T02:50:43.001 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:358: finishing set_chunk to oid 9 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 258048 length: 1323520 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:358: done (15 left) 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:385: set_chunk oid 36 target oid 36 low_tier 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:385: set_chunk oid 36 offset: 577024 length: 1671168 target oid smithi10548823-36 low_tier offset: 577024 snap -1 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:371: finishing set_chunk to oid 22 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 851968 length: 203776 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:374: finishing set_chunk to oid 25 2024-04-20T02:50:43.002 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1440256 length: 412160 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:371: done (15 left) 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:374: done (14 left) 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:386: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:386: set_chunk oid 37 offset: 351232 length: 599552 target oid smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 351232 snap -1 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:387: set_chunk oid 38 target oid 38 low_tier 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:387: set_chunk oid 38 offset: 261120 length: 1307136 target oid smithi10548823-38 low_tier offset: 261120 snap -1 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:381: finishing set_chunk to oid 32 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 449024 length: 1671680 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:384: finishing set_chunk to oid 35 2024-04-20T02:50:43.003 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1246720 length: 767488 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:372: finishing set_chunk to oid 23 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 812544 length: 267264 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:383: finishing set_chunk to oid 34 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 220672 length: 1436672 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:372: done (15 left) 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:381: done (14 left) 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:383: done (13 left) 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:384: done (12 left) 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:388: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:388: set_chunk oid 39 offset: 1005056 length: 186880 target oid smithi10548823-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1005056 snap -1 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:389: set_chunk oid 40 target oid 40 low_tier 2024-04-20T02:50:43.004 INFO:tasks.rados.rados.0.smithi105.stdout:389: set_chunk oid 40 offset: 1066496 length: 242688 target oid smithi10548823-40 low_tier offset: 1066496 snap -1 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:390: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:390: set_chunk oid 41 offset: 925696 length: 1259520 target oid smithi10548823-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 925696 snap -1 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:391: set_chunk oid 42 target oid 42 low_tier 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:391: set_chunk oid 42 offset: 145920 length: 2486272 target oid smithi10548823-42 low_tier offset: 145920 snap -1 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:376: finishing set_chunk to oid 27 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 377344 length: 936448 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:373: finishing set_chunk to oid 24 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 1049600 length: 536576 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:386: finishing set_chunk to oid 37 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 351232 length: 599552 2024-04-20T02:50:43.005 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:373: done (15 left) 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:376: done (14 left) 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:386: done (13 left) 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:392: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:392: set_chunk oid 43 offset: 132096 length: 425984 target oid smithi10548823-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 132096 snap -1 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:393: set_chunk oid 44 target oid 44 low_tier 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:393: set_chunk oid 44 offset: 827904 length: 294912 target oid smithi10548823-44 low_tier offset: 827904 snap -1 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:394: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:394: set_chunk oid 45 offset: 1211904 length: 1249792 target oid smithi10548823-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1211904 snap -1 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:388: finishing set_chunk to oid 39 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1005056 length: 186880 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:380: finishing set_chunk to oid 31 2024-04-20T02:50:43.006 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1459200 length: 518656 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:382: finishing set_chunk to oid 33 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1776640 length: 70656 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:390: finishing set_chunk to oid 41 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 925696 length: 1259520 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:377: finishing set_chunk to oid 28 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 756736 length: 618496 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:391: finishing set_chunk to oid 42 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 145920 length: 2486272 2024-04-20T02:50:43.007 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:387: finishing set_chunk to oid 38 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 261120 length: 1307136 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:377: done (15 left) 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:380: done (14 left) 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:382: done (13 left) 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:387: done (12 left) 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:388: done (11 left) 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:390: done (10 left) 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:391: done (9 left) 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:395: set_chunk oid 46 target oid 46 low_tier 2024-04-20T02:50:43.008 INFO:tasks.rados.rados.0.smithi105.stdout:395: set_chunk oid 46 offset: 146432 length: 2304000 target oid smithi10548823-46 low_tier offset: 146432 snap -1 2024-04-20T02:50:43.009 INFO:tasks.rados.rados.0.smithi105.stdout:396: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.009 INFO:tasks.rados.rados.0.smithi105.stdout:396: set_chunk oid 47 offset: 1629696 length: 613376 target oid smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1629696 snap -1 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:397: set_chunk oid 48 target oid 48 low_tier 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:397: set_chunk oid 48 offset: 1011200 length: 525824 target oid smithi10548823-48 low_tier offset: 1011200 snap -1 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:398: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:398: set_chunk oid 49 offset: 1949696 length: 126976 target oid smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1949696 snap -1 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:399: set_chunk oid 50 target oid 50 low_tier 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:399: set_chunk oid 50 offset: 1031680 length: 750080 target oid smithi10548823-50 low_tier offset: 1031680 snap -1 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:400: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:400: set_chunk oid 1 offset: 930304 length: 79872 target oid smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 930304 snap -1 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:401: set_chunk oid 2 target oid 2 low_tier 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:401: set_chunk oid 2 offset: 499200 length: 1341440 target oid smithi10548823-2 low_tier offset: 499200 snap -1 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:378: finishing set_chunk to oid 29 2024-04-20T02:50:43.010 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 674816 length: 1573888 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:379: finishing set_chunk to oid 30 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 63488 length: 2009088 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:385: finishing set_chunk to oid 36 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 577024 length: 1671168 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:393: finishing set_chunk to oid 44 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 827904 length: 294912 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:394: finishing set_chunk to oid 45 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1211904 length: 1249792 2024-04-20T02:50:43.011 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:389: finishing set_chunk to oid 40 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1066496 length: 242688 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:395: finishing set_chunk to oid 46 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 146432 length: 2304000 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:378: done (15 left) 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:379: done (14 left) 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:385: done (13 left) 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:389: done (12 left) 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:393: done (11 left) 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:394: done (10 left) 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:395: done (9 left) 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:402: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.012 INFO:tasks.rados.rados.0.smithi105.stdout:402: set_chunk oid 3 offset: 731136 length: 930816 target oid smithi10548823-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 731136 snap -1 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:403: set_chunk oid 4 target oid 4 low_tier 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:403: set_chunk oid 4 offset: 333312 length: 542208 target oid smithi10548823-4 low_tier offset: 333312 snap -1 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:404: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:404: set_chunk oid 5 offset: 619520 length: 660480 target oid smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 619520 snap -1 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:405: set_chunk oid 6 target oid 6 low_tier 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:405: set_chunk oid 6 offset: 1620480 length: 430080 target oid smithi10548823-6 low_tier offset: 1620480 snap -1 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:406: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:406: set_chunk oid 7 offset: 384512 length: 1086976 target oid smithi10548823-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 384512 snap -1 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:407: set_chunk oid 8 target oid 8 low_tier 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:407: set_chunk oid 8 offset: 232448 length: 1664512 target oid smithi10548823-8 low_tier offset: 232448 snap -1 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:408: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.013 INFO:tasks.rados.rados.0.smithi105.stdout:408: set_chunk oid 9 offset: 608768 length: 88064 target oid smithi10548823-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 608768 snap -1 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:392: finishing set_chunk to oid 43 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 132096 length: 425984 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:399: finishing set_chunk to oid 50 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1031680 length: 750080 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:396: finishing set_chunk to oid 47 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1629696 length: 613376 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:401: finishing set_chunk to oid 2 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 499200 length: 1341440 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:397: finishing set_chunk to oid 48 2024-04-20T02:50:43.014 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1011200 length: 525824 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:400: finishing set_chunk to oid 1 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 930304 length: 79872 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:404: finishing set_chunk to oid 5 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 619520 length: 660480 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:407: finishing set_chunk to oid 8 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 232448 length: 1664512 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:402: finishing set_chunk to oid 3 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 731136 length: 930816 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:43.015 INFO:tasks.rados.rados.0.smithi105.stdout:403: finishing set_chunk to oid 4 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 333312 length: 542208 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:405: finishing set_chunk to oid 6 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1620480 length: 430080 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:406: finishing set_chunk to oid 7 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 384512 length: 1086976 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:408: finishing set_chunk to oid 9 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 608768 length: 88064 2024-04-20T02:50:43.016 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:392: done (15 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:396: done (14 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:397: done (13 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:399: done (12 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:400: done (11 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:401: done (10 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:402: done (9 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:403: done (8 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:404: done (7 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:405: done (6 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:406: done (5 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:407: done (4 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:408: done (3 left) 2024-04-20T02:50:43.017 INFO:tasks.rados.rados.0.smithi105.stdout:409: set_chunk oid 10 target oid 10 low_tier 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:409: set_chunk oid 10 offset: 1192960 length: 754176 target oid smithi10548823-10 low_tier offset: 1192960 snap -1 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:410: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:410: set_chunk oid 11 offset: 936960 length: 437760 target oid smithi10548823-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 936960 snap -1 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:411: set_chunk oid 12 target oid 12 low_tier 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:411: set_chunk oid 12 offset: 1592832 length: 585728 target oid smithi10548823-12 low_tier offset: 1592832 snap -1 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:411 retry set_chunk ! 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 6 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:409: finishing set_chunk to oid 10 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 1192960 length: 754176 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:409: done (5 left) 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 5 2024-04-20T02:50:43.018 INFO:tasks.rados.rados.0.smithi105.stdout:410: finishing set_chunk to oid 11 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 936960 length: 437760 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:410: done (4 left) 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 4 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:362: finishing set_chunk to oid 13 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:362: done (3 left) 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 3 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:375: finishing set_chunk to oid 26 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 15 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:375: done (2 left) 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 2 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:411: finishing set_chunk to oid 12 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:43.019 INFO:tasks.rados.rados.0.smithi105.stdout:411: done (1 left) 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 1 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:398: finishing set_chunk to oid 49 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:398: done (0 left) 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:412: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:412: set_chunk oid 13 offset: 347136 length: 964608 target oid smithi10548823-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 347136 snap -1 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:413: set_chunk oid 14 target oid 14 low_tier 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:413: set_chunk oid 14 offset: 1161728 length: 365056 target oid smithi10548823-14 low_tier offset: 1161728 snap -1 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:414: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:414: set_chunk oid 15 offset: 441344 length: 1129472 target oid smithi10548823-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 441344 snap -1 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:415: set_chunk oid 16 target oid 16 low_tier 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:415: set_chunk oid 16 offset: 61952 length: 481792 target oid smithi10548823-16 low_tier offset: 61952 snap -1 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:416: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:416: set_chunk oid 17 offset: 361984 length: 1157120 target oid smithi10548823-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 361984 snap -1 2024-04-20T02:50:43.020 INFO:tasks.rados.rados.0.smithi105.stdout:417: set_chunk oid 18 target oid 18 low_tier 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:417: set_chunk oid 18 offset: 1141248 length: 1004544 target oid smithi10548823-18 low_tier offset: 1141248 snap -1 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:418: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:418: set_chunk oid 19 offset: 1163264 length: 400896 target oid smithi10548823-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1163264 snap -1 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:419: set_chunk oid 20 target oid 20 low_tier 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:419: set_chunk oid 20 offset: 226816 length: 1441792 target oid smithi10548823-20 low_tier offset: 226816 snap -1 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:420: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:420: set_chunk oid 21 offset: 1361920 length: 88064 target oid smithi10548823-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1361920 snap -1 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:421: set_chunk oid 22 target oid 22 low_tier 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:421: set_chunk oid 22 offset: 1715200 length: 620032 target oid smithi10548823-22 low_tier offset: 1715200 snap -1 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:422: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:422: set_chunk oid 23 offset: 2097664 length: 103936 target oid smithi10548823-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2097664 snap -1 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:423: set_chunk oid 24 target oid 24 low_tier 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:423: set_chunk oid 24 offset: 508416 length: 747008 target oid smithi10548823-24 low_tier offset: 508416 snap -1 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:424: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.021 INFO:tasks.rados.rados.0.smithi105.stdout:424: set_chunk oid 25 offset: 216576 length: 831488 target oid smithi10548823-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 216576 snap -1 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:425: set_chunk oid 26 target oid 26 low_tier 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:425: set_chunk oid 26 offset: 229376 length: 803840 target oid smithi10548823-26 low_tier offset: 229376 snap -1 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:426: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:426: set_chunk oid 27 offset: 1948160 length: 308736 target oid smithi10548823-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1948160 snap -1 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:427: set_chunk oid 28 target oid 28 low_tier 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:427: set_chunk oid 28 offset: 1071104 length: 1377280 target oid smithi10548823-28 low_tier offset: 1071104 snap -1 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:412: finishing set_chunk to oid 13 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 347136 length: 964608 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:417: finishing set_chunk to oid 18 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1141248 length: 1004544 2024-04-20T02:50:43.022 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:412: done (15 left) 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:417: done (14 left) 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:428: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:428: set_chunk oid 29 offset: 408576 length: 141312 target oid smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 408576 snap -1 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:429: set_chunk oid 30 target oid 30 low_tier 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:429: set_chunk oid 30 offset: 1568768 length: 684544 target oid smithi10548823-30 low_tier offset: 1568768 snap -1 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:414: finishing set_chunk to oid 15 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 441344 length: 1129472 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:414: done (15 left) 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:430: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout:430: set_chunk oid 31 offset: 114176 length: 1336320 target oid smithi10548823-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 114176 snap -1 2024-04-20T02:50:43.023 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:418: finishing set_chunk to oid 19 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1163264 length: 400896 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:418: done (15 left) 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:431: set_chunk oid 32 target oid 32 low_tier 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:431: set_chunk oid 32 offset: 67584 length: 931328 target oid smithi10548823-32 low_tier offset: 67584 snap -1 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:416: finishing set_chunk to oid 17 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 361984 length: 1157120 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:413: finishing set_chunk to oid 14 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 1161728 length: 365056 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:423: finishing set_chunk to oid 24 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 508416 length: 747008 2024-04-20T02:50:43.024 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:420: finishing set_chunk to oid 21 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1361920 length: 88064 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:428: finishing set_chunk to oid 29 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 408576 length: 141312 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:421: finishing set_chunk to oid 22 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1715200 length: 620032 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:419: finishing set_chunk to oid 20 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 226816 length: 1441792 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:413: done (15 left) 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:416: done (14 left) 2024-04-20T02:50:43.025 INFO:tasks.rados.rados.0.smithi105.stdout:419: done (13 left) 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:420: done (12 left) 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:421: done (11 left) 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:423: done (10 left) 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:428: done (9 left) 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:432: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:432: set_chunk oid 33 offset: 59904 length: 533504 target oid smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 59904 snap -1 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:433: set_chunk oid 34 target oid 34 low_tier 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:433: set_chunk oid 34 offset: 2015744 length: 202240 target oid smithi10548823-34 low_tier offset: 2015744 snap -1 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:434: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:434: set_chunk oid 35 offset: 1186816 length: 1075712 target oid smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1186816 snap -1 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:435: set_chunk oid 36 target oid 36 low_tier 2024-04-20T02:50:43.026 INFO:tasks.rados.rados.0.smithi105.stdout:435: set_chunk oid 36 offset: 252928 length: 2199552 target oid smithi10548823-36 low_tier offset: 252928 snap -1 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:436: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:436: set_chunk oid 37 offset: 2368512 length: 3072 target oid smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2368512 snap -1 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:437: set_chunk oid 38 target oid 38 low_tier 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:437: set_chunk oid 38 offset: 477696 length: 1526784 target oid smithi10548823-38 low_tier offset: 477696 snap -1 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:438: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:438: set_chunk oid 39 offset: 364544 length: 2203136 target oid smithi10548823-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 364544 snap -1 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:425: finishing set_chunk to oid 26 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 229376 length: 803840 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:430: finishing set_chunk to oid 31 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 114176 length: 1336320 2024-04-20T02:50:43.027 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:427: finishing set_chunk to oid 28 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1071104 length: 1377280 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:429: finishing set_chunk to oid 30 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 1568768 length: 684544 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:424: finishing set_chunk to oid 25 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 216576 length: 831488 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:431: finishing set_chunk to oid 32 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 67584 length: 931328 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:438: finishing set_chunk to oid 39 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 364544 length: 2203136 2024-04-20T02:50:43.028 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:424: done (15 left) 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:425: done (14 left) 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:427: done (13 left) 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:429: done (12 left) 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:430: done (11 left) 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:431: done (10 left) 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:438: done (9 left) 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:439: set_chunk oid 40 target oid 40 low_tier 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:439: set_chunk oid 40 offset: 7168 length: 377344 target oid smithi10548823-40 low_tier offset: 7168 snap -1 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:440: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:440: set_chunk oid 41 offset: 195072 length: 2406400 target oid smithi10548823-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 195072 snap -1 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:441: set_chunk oid 42 target oid 42 low_tier 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:441: set_chunk oid 42 offset: 703488 length: 1444352 target oid smithi10548823-42 low_tier offset: 703488 snap -1 2024-04-20T02:50:43.029 INFO:tasks.rados.rados.0.smithi105.stdout:442: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:442: set_chunk oid 43 offset: 55808 length: 1228800 target oid smithi10548823-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 55808 snap -1 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:443: set_chunk oid 44 target oid 44 low_tier 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:443: set_chunk oid 44 offset: 675840 length: 1241600 target oid smithi10548823-44 low_tier offset: 675840 snap -1 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:444: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:444: set_chunk oid 45 offset: 1104384 length: 193536 target oid smithi10548823-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1104384 snap -1 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:445: set_chunk oid 46 target oid 46 low_tier 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:445: set_chunk oid 46 offset: 1090560 length: 528896 target oid smithi10548823-46 low_tier offset: 1090560 snap -1 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:426: finishing set_chunk to oid 27 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1948160 length: 308736 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:434: finishing set_chunk to oid 35 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1186816 length: 1075712 2024-04-20T02:50:43.030 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:435: finishing set_chunk to oid 36 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 252928 length: 2199552 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:432: finishing set_chunk to oid 33 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 59904 length: 533504 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:426: done (15 left) 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:432: done (14 left) 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:434: done (13 left) 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:435: done (12 left) 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:446: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:446: set_chunk oid 47 offset: 1497600 length: 1077760 target oid smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1497600 snap -1 2024-04-20T02:50:43.031 INFO:tasks.rados.rados.0.smithi105.stdout:447: set_chunk oid 48 target oid 48 low_tier 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:447: set_chunk oid 48 offset: 1021440 length: 1160704 target oid smithi10548823-48 low_tier offset: 1021440 snap -1 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:448: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:448: set_chunk oid 49 offset: 567296 length: 1675776 target oid smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 567296 snap -1 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:449: set_chunk oid 50 target oid 50 low_tier 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:449: set_chunk oid 50 offset: 1347072 length: 1409024 target oid smithi10548823-50 low_tier offset: 1347072 snap -1 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:442: finishing set_chunk to oid 43 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 55808 length: 1228800 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:442: done (15 left) 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:450: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:450: set_chunk oid 1 offset: 391168 length: 289280 target oid smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 391168 snap -1 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.032 INFO:tasks.rados.rados.0.smithi105.stdout:443: finishing set_chunk to oid 44 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 675840 length: 1241600 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:444: finishing set_chunk to oid 45 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1104384 length: 193536 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:443: done (15 left) 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:444: done (14 left) 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:451: set_chunk oid 2 target oid 2 low_tier 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:451: set_chunk oid 2 offset: 1315328 length: 546816 target oid smithi10548823-2 low_tier offset: 1315328 snap -1 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:452: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:452: set_chunk oid 3 offset: 401408 length: 1344512 target oid smithi10548823-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 401408 snap -1 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:448: finishing set_chunk to oid 49 2024-04-20T02:50:43.033 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 567296 length: 1675776 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:441: finishing set_chunk to oid 42 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 703488 length: 1444352 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:437: finishing set_chunk to oid 38 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 477696 length: 1526784 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:446: finishing set_chunk to oid 47 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1497600 length: 1077760 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:449: finishing set_chunk to oid 50 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1347072 length: 1409024 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:43.034 INFO:tasks.rados.rados.0.smithi105.stdout:440: finishing set_chunk to oid 41 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 195072 length: 2406400 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:437: done (15 left) 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:440: done (14 left) 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:441: done (13 left) 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:446: done (12 left) 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:448: done (11 left) 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:449: done (10 left) 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:453: set_chunk oid 4 target oid 4 low_tier 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:453: set_chunk oid 4 offset: 318464 length: 1260032 target oid smithi10548823-4 low_tier offset: 318464 snap -1 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:454: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:454: set_chunk oid 5 offset: 419840 length: 1492992 target oid smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 419840 snap -1 2024-04-20T02:50:43.035 INFO:tasks.rados.rados.0.smithi105.stdout:455: set_chunk oid 6 target oid 6 low_tier 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:455: set_chunk oid 6 offset: 527360 length: 1102336 target oid smithi10548823-6 low_tier offset: 527360 snap -1 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:456: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:456: set_chunk oid 7 offset: 347136 length: 811520 target oid smithi10548823-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 347136 snap -1 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:457: set_chunk oid 8 target oid 8 low_tier 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:457: set_chunk oid 8 offset: 504320 length: 220672 target oid smithi10548823-8 low_tier offset: 504320 snap -1 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:458: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:458: set_chunk oid 9 offset: 1142272 length: 34304 target oid smithi10548823-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1142272 snap -1 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:450: finishing set_chunk to oid 1 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 391168 length: 289280 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:451: finishing set_chunk to oid 2 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1315328 length: 546816 2024-04-20T02:50:43.036 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:452: finishing set_chunk to oid 3 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 401408 length: 1344512 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:450: done (15 left) 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:451: done (14 left) 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:452: done (13 left) 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:459: set_chunk oid 10 target oid 10 low_tier 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:459: set_chunk oid 10 offset: 1134080 length: 431104 target oid smithi10548823-10 low_tier offset: 1134080 snap -1 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:460: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:460: set_chunk oid 11 offset: 267264 length: 339456 target oid smithi10548823-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 267264 snap -1 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:461: set_chunk oid 12 target oid 12 low_tier 2024-04-20T02:50:43.037 INFO:tasks.rados.rados.0.smithi105.stdout:461: set_chunk oid 12 offset: 1470976 length: 70656 target oid smithi10548823-12 low_tier offset: 1470976 snap -1 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:447: finishing set_chunk to oid 48 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1021440 length: 1160704 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:447: done (15 left) 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:462: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:462: set_chunk oid 13 offset: 14336 length: 1973248 target oid smithi10548823-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 14336 snap -1 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:454: finishing set_chunk to oid 5 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 419840 length: 1492992 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:457: finishing set_chunk to oid 8 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 504320 length: 220672 2024-04-20T02:50:43.038 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:454: done (15 left) 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:457: done (14 left) 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:463: set_chunk oid 14 target oid 14 low_tier 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:463: set_chunk oid 14 offset: 991232 length: 1155072 target oid smithi10548823-14 low_tier offset: 991232 snap -1 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:464: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:464: set_chunk oid 15 offset: 806400 length: 213504 target oid smithi10548823-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 806400 snap -1 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:453: finishing set_chunk to oid 4 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 318464 length: 1260032 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:455: finishing set_chunk to oid 6 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 527360 length: 1102336 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:456: finishing set_chunk to oid 7 2024-04-20T02:50:43.039 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 347136 length: 811520 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:453: done (15 left) 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:455: done (14 left) 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:456: done (13 left) 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:464 retry set_chunk ! 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 13 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:459: finishing set_chunk to oid 10 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 1134080 length: 431104 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:459: done (12 left) 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 12 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:458: finishing set_chunk to oid 9 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1142272 length: 34304 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:43.040 INFO:tasks.rados.rados.0.smithi105.stdout:460: finishing set_chunk to oid 11 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 267264 length: 339456 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:458: done (11 left) 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:460: done (10 left) 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 10 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:462: finishing set_chunk to oid 13 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 14336 length: 1973248 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:462: done (9 left) 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 9 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:464: finishing set_chunk to oid 15 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 806400 length: 213504 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:43.041 INFO:tasks.rados.rados.0.smithi105.stdout:464: done (8 left) 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 8 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout:463: finishing set_chunk to oid 14 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 991232 length: 1155072 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout:463: done (7 left) 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 7 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout:445: finishing set_chunk to oid 46 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1090560 length: 528896 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout:461: finishing set_chunk to oid 12 2024-04-20T02:50:43.042 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1470976 length: 70656 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:445: done (6 left) 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:461: done (5 left) 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 5 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:415: finishing set_chunk to oid 16 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 9 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:415: done (4 left) 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 4 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:422: finishing set_chunk to oid 23 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:422: done (3 left) 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 3 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:433: finishing set_chunk to oid 34 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:43.043 INFO:tasks.rados.rados.0.smithi105.stdout:433: done (2 left) 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 2 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:436: finishing set_chunk to oid 37 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:439: finishing set_chunk to oid 40 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 21 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:436: done (1 left) 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:439: done (0 left) 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:465: set_chunk oid 16 target oid 16 low_tier 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:465: set_chunk oid 16 offset: 885760 length: 212992 target oid smithi10548823-16 low_tier offset: 885760 snap -1 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:466: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:466: set_chunk oid 17 offset: 1495040 length: 611328 target oid smithi10548823-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1495040 snap -1 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:467: set_chunk oid 18 target oid 18 low_tier 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:467: set_chunk oid 18 offset: 247296 length: 108544 target oid smithi10548823-18 low_tier offset: 247296 snap -1 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:468: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.044 INFO:tasks.rados.rados.0.smithi105.stdout:468: set_chunk oid 19 offset: 208384 length: 196608 target oid smithi10548823-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 208384 snap -1 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:469: set_chunk oid 20 target oid 20 low_tier 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:469: set_chunk oid 20 offset: 642048 length: 1248768 target oid smithi10548823-20 low_tier offset: 642048 snap -1 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:470: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:470: set_chunk oid 21 offset: 25088 length: 1049600 target oid smithi10548823-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 25088 snap -1 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:471: set_chunk oid 22 target oid 22 low_tier 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:471: set_chunk oid 22 offset: 178176 length: 875520 target oid smithi10548823-22 low_tier offset: 178176 snap -1 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:472: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:472: set_chunk oid 23 offset: 327680 length: 843776 target oid smithi10548823-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 327680 snap -1 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:473: set_chunk oid 24 target oid 24 low_tier 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:473: set_chunk oid 24 offset: 963072 length: 596480 target oid smithi10548823-24 low_tier offset: 963072 snap -1 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:474: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:474: set_chunk oid 25 offset: 1577472 length: 541184 target oid smithi10548823-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1577472 snap -1 2024-04-20T02:50:43.045 INFO:tasks.rados.rados.0.smithi105.stdout:475: set_chunk oid 26 target oid 26 low_tier 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:475: set_chunk oid 26 offset: 1277440 length: 821760 target oid smithi10548823-26 low_tier offset: 1277440 snap -1 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:476: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:476: set_chunk oid 27 offset: 1893888 length: 286720 target oid smithi10548823-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1893888 snap -1 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:477: set_chunk oid 28 target oid 28 low_tier 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:477: set_chunk oid 28 offset: 155648 length: 616448 target oid smithi10548823-28 low_tier offset: 155648 snap -1 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:478: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:478: set_chunk oid 29 offset: 684032 length: 1622528 target oid smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 684032 snap -1 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:479: set_chunk oid 30 target oid 30 low_tier 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:479: set_chunk oid 30 offset: 725504 length: 776192 target oid smithi10548823-30 low_tier offset: 725504 snap -1 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:480: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:480: set_chunk oid 31 offset: 1009152 length: 1030656 target oid smithi10548823-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1009152 snap -1 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:466: finishing set_chunk to oid 17 2024-04-20T02:50:43.046 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1495040 length: 611328 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:466: done (15 left) 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:481: set_chunk oid 32 target oid 32 low_tier 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:481: set_chunk oid 32 offset: 1095168 length: 1113600 target oid smithi10548823-32 low_tier offset: 1095168 snap -1 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:465: finishing set_chunk to oid 16 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 885760 length: 212992 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:469: finishing set_chunk to oid 20 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 642048 length: 1248768 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:467: finishing set_chunk to oid 18 2024-04-20T02:50:43.047 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 247296 length: 108544 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:471: finishing set_chunk to oid 22 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 178176 length: 875520 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:473: finishing set_chunk to oid 24 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 963072 length: 596480 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:470: finishing set_chunk to oid 21 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 25088 length: 1049600 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:465: done (15 left) 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:467: done (14 left) 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:469: done (13 left) 2024-04-20T02:50:43.048 INFO:tasks.rados.rados.0.smithi105.stdout:470: done (12 left) 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:471: done (11 left) 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:473: done (10 left) 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:482: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:482: set_chunk oid 33 offset: 296448 length: 1163776 target oid smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 296448 snap -1 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:483: set_chunk oid 34 target oid 34 low_tier 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:483: set_chunk oid 34 offset: 1114624 length: 583680 target oid smithi10548823-34 low_tier offset: 1114624 snap -1 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:484: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:484: set_chunk oid 35 offset: 1298944 length: 1099264 target oid smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1298944 snap -1 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:485: set_chunk oid 36 target oid 36 low_tier 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:485: set_chunk oid 36 offset: 654848 length: 526336 target oid smithi10548823-36 low_tier offset: 654848 snap -1 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:486: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:486: set_chunk oid 37 offset: 1998848 length: 492032 target oid smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1998848 snap -1 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:487: set_chunk oid 38 target oid 38 low_tier 2024-04-20T02:50:43.049 INFO:tasks.rados.rados.0.smithi105.stdout:487: set_chunk oid 38 offset: 1825280 length: 72704 target oid smithi10548823-38 low_tier offset: 1825280 snap -1 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:468: finishing set_chunk to oid 19 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 208384 length: 196608 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:481: finishing set_chunk to oid 32 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1095168 length: 1113600 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:472: finishing set_chunk to oid 23 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 327680 length: 843776 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:474: finishing set_chunk to oid 25 2024-04-20T02:50:43.050 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1577472 length: 541184 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:475: finishing set_chunk to oid 26 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1277440 length: 821760 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:468: done (15 left) 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:472: done (14 left) 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:474: done (13 left) 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:475: done (12 left) 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:481: done (11 left) 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:488: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:488: set_chunk oid 39 offset: 1077248 length: 1422336 target oid smithi10548823-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1077248 snap -1 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:489: set_chunk oid 40 target oid 40 low_tier 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:489: set_chunk oid 40 offset: 262656 length: 2304512 target oid smithi10548823-40 low_tier offset: 262656 snap -1 2024-04-20T02:50:43.051 INFO:tasks.rados.rados.0.smithi105.stdout:490: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:490: set_chunk oid 41 offset: 507392 length: 1451008 target oid smithi10548823-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 507392 snap -1 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:491: set_chunk oid 42 target oid 42 low_tier 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:491: set_chunk oid 42 offset: 415744 length: 1104896 target oid smithi10548823-42 low_tier offset: 415744 snap -1 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:492: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:492: set_chunk oid 43 offset: 89600 length: 2413568 target oid smithi10548823-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 89600 snap -1 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:478: finishing set_chunk to oid 29 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 684032 length: 1622528 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:484: finishing set_chunk to oid 35 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1298944 length: 1099264 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:43.052 INFO:tasks.rados.rados.0.smithi105.stdout:479: finishing set_chunk to oid 30 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 725504 length: 776192 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:480: finishing set_chunk to oid 31 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1009152 length: 1030656 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:477: finishing set_chunk to oid 28 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 155648 length: 616448 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:482: finishing set_chunk to oid 33 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 296448 length: 1163776 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:485: finishing set_chunk to oid 36 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 654848 length: 526336 2024-04-20T02:50:43.053 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:483: finishing set_chunk to oid 34 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1114624 length: 583680 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:476: finishing set_chunk to oid 27 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1893888 length: 286720 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:486: finishing set_chunk to oid 37 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1998848 length: 492032 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:476: done (15 left) 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:477: done (14 left) 2024-04-20T02:50:43.054 INFO:tasks.rados.rados.0.smithi105.stdout:478: done (13 left) 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:479: done (12 left) 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:480: done (11 left) 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:482: done (10 left) 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:483: done (9 left) 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:484: done (8 left) 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:485: done (7 left) 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:486: done (6 left) 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:493: set_chunk oid 44 target oid 44 low_tier 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:493: set_chunk oid 44 offset: 1350144 length: 1174016 target oid smithi10548823-44 low_tier offset: 1350144 snap -1 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:494: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:494: set_chunk oid 45 offset: 437760 length: 376832 target oid smithi10548823-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 437760 snap -1 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:495: set_chunk oid 46 target oid 46 low_tier 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:495: set_chunk oid 46 offset: 1215488 length: 224768 target oid smithi10548823-46 low_tier offset: 1215488 snap -1 2024-04-20T02:50:43.055 INFO:tasks.rados.rados.0.smithi105.stdout:496: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:496: set_chunk oid 47 offset: 459776 length: 390144 target oid smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 459776 snap -1 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:497: set_chunk oid 48 target oid 48 low_tier 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:497: set_chunk oid 48 offset: 1254400 length: 1447424 target oid smithi10548823-48 low_tier offset: 1254400 snap -1 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:498: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:498: set_chunk oid 49 offset: 572928 length: 742400 target oid smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 572928 snap -1 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:499: set_chunk oid 50 target oid 50 low_tier 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:499: set_chunk oid 50 offset: 148992 length: 1094144 target oid smithi10548823-50 low_tier offset: 148992 snap -1 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:500: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:500: set_chunk oid 1 offset: 1620480 length: 378368 target oid smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1620480 snap -1 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:501: set_chunk oid 2 target oid 2 low_tier 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:501: set_chunk oid 2 offset: 1328128 length: 268800 target oid smithi10548823-2 low_tier offset: 1328128 snap -1 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:502: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout:502: set_chunk oid 3 offset: 978944 length: 914432 target oid smithi10548823-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 978944 snap -1 2024-04-20T02:50:43.056 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:487: finishing set_chunk to oid 38 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 1825280 length: 72704 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:491: finishing set_chunk to oid 42 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 415744 length: 1104896 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:488: finishing set_chunk to oid 39 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1077248 length: 1422336 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:489: finishing set_chunk to oid 40 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 262656 length: 2304512 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:490: finishing set_chunk to oid 41 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 507392 length: 1451008 2024-04-20T02:50:43.057 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:487: done (15 left) 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:488: done (14 left) 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:489: done (13 left) 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:490: done (12 left) 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:491: done (11 left) 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:503: set_chunk oid 4 target oid 4 low_tier 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:503: set_chunk oid 4 offset: 927232 length: 660480 target oid smithi10548823-4 low_tier offset: 927232 snap -1 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:504: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:504: set_chunk oid 5 offset: 940544 length: 206848 target oid smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 940544 snap -1 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:505: set_chunk oid 6 target oid 6 low_tier 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:505: set_chunk oid 6 offset: 823296 length: 219648 target oid smithi10548823-6 low_tier offset: 823296 snap -1 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:506: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:506: set_chunk oid 7 offset: 1136640 length: 542208 target oid smithi10548823-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1136640 snap -1 2024-04-20T02:50:43.058 INFO:tasks.rados.rados.0.smithi105.stdout:507: set_chunk oid 8 target oid 8 low_tier 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:507: set_chunk oid 8 offset: 251904 length: 1738752 target oid smithi10548823-8 low_tier offset: 251904 snap -1 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:493: finishing set_chunk to oid 44 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 1350144 length: 1174016 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:494: finishing set_chunk to oid 45 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 437760 length: 376832 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:495: finishing set_chunk to oid 46 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1215488 length: 224768 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:499: finishing set_chunk to oid 50 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 148992 length: 1094144 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:43.059 INFO:tasks.rados.rados.0.smithi105.stdout:496: finishing set_chunk to oid 47 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 459776 length: 390144 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:493: done (15 left) 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:494: done (14 left) 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:495: done (13 left) 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:496: done (12 left) 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:499: done (11 left) 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:508: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:508: set_chunk oid 9 offset: 591360 length: 1195008 target oid smithi10548823-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 591360 snap -1 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:509: set_chunk oid 10 target oid 10 low_tier 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:509: set_chunk oid 10 offset: 619008 length: 99328 target oid smithi10548823-10 low_tier offset: 619008 snap -1 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:510: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:510: set_chunk oid 11 offset: 86528 length: 801280 target oid smithi10548823-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 86528 snap -1 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:511: set_chunk oid 12 target oid 12 low_tier 2024-04-20T02:50:43.060 INFO:tasks.rados.rados.0.smithi105.stdout:511: set_chunk oid 12 offset: 2029056 length: 116736 target oid smithi10548823-12 low_tier offset: 2029056 snap -1 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:512: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:512: set_chunk oid 13 offset: 545280 length: 1413120 target oid smithi10548823-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 545280 snap -1 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:502: finishing set_chunk to oid 3 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 978944 length: 914432 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:498: finishing set_chunk to oid 49 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 572928 length: 742400 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:501: finishing set_chunk to oid 2 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1328128 length: 268800 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:492: finishing set_chunk to oid 43 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 89600 length: 2413568 2024-04-20T02:50:43.061 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:492: done (15 left) 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:498: done (14 left) 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:501: done (13 left) 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:502: done (12 left) 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:513: set_chunk oid 14 target oid 14 low_tier 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:513: set_chunk oid 14 offset: 1463296 length: 147456 target oid smithi10548823-14 low_tier offset: 1463296 snap -1 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:514: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:514: set_chunk oid 15 offset: 828928 length: 430080 target oid smithi10548823-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 828928 snap -1 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:515: set_chunk oid 16 target oid 16 low_tier 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:515: set_chunk oid 16 offset: 1507328 length: 474624 target oid smithi10548823-16 low_tier offset: 1507328 snap -1 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:516: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:516: set_chunk oid 17 offset: 2018304 length: 192000 target oid smithi10548823-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2018304 snap -1 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.062 INFO:tasks.rados.rados.0.smithi105.stdout:497: finishing set_chunk to oid 48 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1254400 length: 1447424 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:500: finishing set_chunk to oid 1 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1620480 length: 378368 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:497: done (15 left) 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:500: done (14 left) 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:517: set_chunk oid 18 target oid 18 low_tier 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:517: set_chunk oid 18 offset: 1708544 length: 354304 target oid smithi10548823-18 low_tier offset: 1708544 snap -1 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:518: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:518: set_chunk oid 19 offset: 128000 length: 1107456 target oid smithi10548823-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 128000 snap -1 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:507: finishing set_chunk to oid 8 2024-04-20T02:50:43.063 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 251904 length: 1738752 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:504: finishing set_chunk to oid 5 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 940544 length: 206848 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:512: finishing set_chunk to oid 13 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 545280 length: 1413120 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:503: finishing set_chunk to oid 4 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 927232 length: 660480 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:505: finishing set_chunk to oid 6 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 823296 length: 219648 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:510: finishing set_chunk to oid 11 2024-04-20T02:50:43.064 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 86528 length: 801280 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:503: done (15 left) 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:504: done (14 left) 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:505: done (13 left) 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:507: done (12 left) 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:510: done (11 left) 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:512: done (10 left) 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:519: set_chunk oid 20 target oid 20 low_tier 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:519: set_chunk oid 20 offset: 450048 length: 84480 target oid smithi10548823-20 low_tier offset: 450048 snap -1 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:520: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:520: set_chunk oid 21 offset: 758272 length: 964096 target oid smithi10548823-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 758272 snap -1 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:521: set_chunk oid 22 target oid 22 low_tier 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:521: set_chunk oid 22 offset: 928256 length: 1127936 target oid smithi10548823-22 low_tier offset: 928256 snap -1 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:522: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:522: set_chunk oid 23 offset: 1276928 length: 554496 target oid smithi10548823-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1276928 snap -1 2024-04-20T02:50:43.065 INFO:tasks.rados.rados.0.smithi105.stdout:523: set_chunk oid 24 target oid 24 low_tier 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:523: set_chunk oid 24 offset: 471040 length: 764928 target oid smithi10548823-24 low_tier offset: 471040 snap -1 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:524: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:524: set_chunk oid 25 offset: 735744 length: 1489408 target oid smithi10548823-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 735744 snap -1 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:506: finishing set_chunk to oid 7 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1136640 length: 542208 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:513: finishing set_chunk to oid 14 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 1463296 length: 147456 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:511: finishing set_chunk to oid 12 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 2029056 length: 116736 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:43.066 INFO:tasks.rados.rados.0.smithi105.stdout:514: finishing set_chunk to oid 15 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 828928 length: 430080 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:517: finishing set_chunk to oid 18 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1708544 length: 354304 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:508: finishing set_chunk to oid 9 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 591360 length: 1195008 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:518: finishing set_chunk to oid 19 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 128000 length: 1107456 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:523: finishing set_chunk to oid 24 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 471040 length: 764928 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:43.067 INFO:tasks.rados.rados.0.smithi105.stdout:506: done (15 left) 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:508: done (14 left) 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:511: done (13 left) 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:513: done (12 left) 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:514: done (11 left) 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:517: done (10 left) 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:518: done (9 left) 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:523: done (8 left) 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:525: set_chunk oid 26 target oid 26 low_tier 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:525: set_chunk oid 26 offset: 356864 length: 35840 target oid smithi10548823-26 low_tier offset: 356864 snap -1 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:526: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:526: set_chunk oid 27 offset: 2024448 length: 251904 target oid smithi10548823-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2024448 snap -1 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:527: set_chunk oid 28 target oid 28 low_tier 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:527: set_chunk oid 28 offset: 1274368 length: 1151488 target oid smithi10548823-28 low_tier offset: 1274368 snap -1 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:528: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.068 INFO:tasks.rados.rados.0.smithi105.stdout:528: set_chunk oid 29 offset: 1291264 length: 864256 target oid smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1291264 snap -1 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:529: set_chunk oid 30 target oid 30 low_tier 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:529: set_chunk oid 30 offset: 81408 length: 215552 target oid smithi10548823-30 low_tier offset: 81408 snap -1 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:530: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:530: set_chunk oid 31 offset: 797696 length: 1120256 target oid smithi10548823-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 797696 snap -1 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:531: set_chunk oid 32 target oid 32 low_tier 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:531: set_chunk oid 32 offset: 72192 length: 396288 target oid smithi10548823-32 low_tier offset: 72192 snap -1 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:532: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:532: set_chunk oid 33 offset: 684544 length: 1547264 target oid smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 684544 snap -1 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:519: finishing set_chunk to oid 20 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 450048 length: 84480 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:519: done (15 left) 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:533: set_chunk oid 34 target oid 34 low_tier 2024-04-20T02:50:43.069 INFO:tasks.rados.rados.0.smithi105.stdout:533: set_chunk oid 34 offset: 1090048 length: 1136640 target oid smithi10548823-34 low_tier offset: 1090048 snap -1 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:520: finishing set_chunk to oid 21 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 758272 length: 964096 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:520: done (15 left) 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:534: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:534: set_chunk oid 35 offset: 1794048 length: 701440 target oid smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1794048 snap -1 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:521: finishing set_chunk to oid 22 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 928256 length: 1127936 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:524: finishing set_chunk to oid 25 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 735744 length: 1489408 2024-04-20T02:50:43.070 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:522: finishing set_chunk to oid 23 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1276928 length: 554496 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:533: finishing set_chunk to oid 34 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1090048 length: 1136640 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:527: finishing set_chunk to oid 28 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1274368 length: 1151488 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:530: finishing set_chunk to oid 31 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 797696 length: 1120256 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:532: finishing set_chunk to oid 33 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 684544 length: 1547264 2024-04-20T02:50:43.071 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:528: finishing set_chunk to oid 29 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1291264 length: 864256 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:521: done (15 left) 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:522: done (14 left) 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:524: done (13 left) 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:527: done (12 left) 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:528: done (11 left) 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:530: done (10 left) 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:532: done (9 left) 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:533: done (8 left) 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:535: set_chunk oid 36 target oid 36 low_tier 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:535: set_chunk oid 36 offset: 487424 length: 1326080 target oid smithi10548823-36 low_tier offset: 487424 snap -1 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:536: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.072 INFO:tasks.rados.rados.0.smithi105.stdout:536: set_chunk oid 37 offset: 475136 length: 43008 target oid smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 475136 snap -1 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:537: set_chunk oid 38 target oid 38 low_tier 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:537: set_chunk oid 38 offset: 329728 length: 1543680 target oid smithi10548823-38 low_tier offset: 329728 snap -1 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:538: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:538: set_chunk oid 39 offset: 1576448 length: 110080 target oid smithi10548823-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1576448 snap -1 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:539: set_chunk oid 40 target oid 40 low_tier 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:539: set_chunk oid 40 offset: 1906176 length: 82944 target oid smithi10548823-40 low_tier offset: 1906176 snap -1 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:540: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:540: set_chunk oid 41 offset: 486912 length: 1009664 target oid smithi10548823-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 486912 snap -1 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:541: set_chunk oid 42 target oid 42 low_tier 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:541: set_chunk oid 42 offset: 527872 length: 894976 target oid smithi10548823-42 low_tier offset: 527872 snap -1 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:542: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:542: set_chunk oid 43 offset: 136192 length: 2046464 target oid smithi10548823-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 136192 snap -1 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:538: finishing set_chunk to oid 39 2024-04-20T02:50:43.073 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1576448 length: 110080 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:534: finishing set_chunk to oid 35 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1794048 length: 701440 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:536: finishing set_chunk to oid 37 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 475136 length: 43008 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:535: finishing set_chunk to oid 36 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 487424 length: 1326080 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:537: finishing set_chunk to oid 38 2024-04-20T02:50:43.074 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 329728 length: 1543680 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:540: finishing set_chunk to oid 41 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 486912 length: 1009664 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:534: done (15 left) 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:535: done (14 left) 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:536: done (13 left) 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:537: done (12 left) 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:538: done (11 left) 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:540: done (10 left) 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:543: set_chunk oid 44 target oid 44 low_tier 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:543: set_chunk oid 44 offset: 132096 length: 278016 target oid smithi10548823-44 low_tier offset: 132096 snap -1 2024-04-20T02:50:43.075 INFO:tasks.rados.rados.0.smithi105.stdout:544: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:544: set_chunk oid 45 offset: 634368 length: 236544 target oid smithi10548823-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 634368 snap -1 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:545: set_chunk oid 46 target oid 46 low_tier 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:545: set_chunk oid 46 offset: 422912 length: 1384960 target oid smithi10548823-46 low_tier offset: 422912 snap -1 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:546: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:546: set_chunk oid 47 offset: 1357824 length: 40960 target oid smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1357824 snap -1 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:547: set_chunk oid 48 target oid 48 low_tier 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:547: set_chunk oid 48 offset: 1610752 length: 619008 target oid smithi10548823-48 low_tier offset: 1610752 snap -1 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:548: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:548: set_chunk oid 49 offset: 543744 length: 1011200 target oid smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 543744 snap -1 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:542: finishing set_chunk to oid 43 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 136192 length: 2046464 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:43.076 INFO:tasks.rados.rados.0.smithi105.stdout:541: finishing set_chunk to oid 42 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 527872 length: 894976 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:541: done (15 left) 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:542: done (14 left) 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:549: set_chunk oid 50 target oid 50 low_tier 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:549: set_chunk oid 50 offset: 652288 length: 796160 target oid smithi10548823-50 low_tier offset: 652288 snap -1 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:550: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:550: set_chunk oid 1 offset: 190464 length: 237568 target oid smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 190464 snap -1 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:543: finishing set_chunk to oid 44 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 132096 length: 278016 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:544: finishing set_chunk to oid 45 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 634368 length: 236544 2024-04-20T02:50:43.077 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:546: finishing set_chunk to oid 47 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1357824 length: 40960 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:545: finishing set_chunk to oid 46 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 422912 length: 1384960 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:548: finishing set_chunk to oid 49 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 543744 length: 1011200 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:543: done (15 left) 2024-04-20T02:50:43.078 INFO:tasks.rados.rados.0.smithi105.stdout:544: done (14 left) 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:545: done (13 left) 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:546: done (12 left) 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:548: done (11 left) 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:551: set_chunk oid 2 target oid 2 low_tier 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:551: set_chunk oid 2 offset: 921600 length: 709120 target oid smithi10548823-2 low_tier offset: 921600 snap -1 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:552: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:552: set_chunk oid 3 offset: 1486848 length: 424448 target oid smithi10548823-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1486848 snap -1 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:553: set_chunk oid 4 target oid 4 low_tier 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:553: set_chunk oid 4 offset: 914944 length: 1053184 target oid smithi10548823-4 low_tier offset: 914944 snap -1 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:554: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:554: set_chunk oid 5 offset: 803840 length: 974848 target oid smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 803840 snap -1 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:555: set_chunk oid 6 target oid 6 low_tier 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:555: set_chunk oid 6 offset: 832512 length: 317952 target oid smithi10548823-6 low_tier offset: 832512 snap -1 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.079 INFO:tasks.rados.rados.0.smithi105.stdout:547: finishing set_chunk to oid 48 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1610752 length: 619008 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:549: finishing set_chunk to oid 50 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 652288 length: 796160 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:547: done (15 left) 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:549: done (14 left) 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:556: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:556: set_chunk oid 7 offset: 746496 length: 498688 target oid smithi10548823-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 746496 snap -1 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:557: set_chunk oid 8 target oid 8 low_tier 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:557: set_chunk oid 8 offset: 118272 length: 1825792 target oid smithi10548823-8 low_tier offset: 118272 snap -1 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:551: finishing set_chunk to oid 2 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 921600 length: 709120 2024-04-20T02:50:43.080 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:551: done (15 left) 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:558: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:558: set_chunk oid 9 offset: 322560 length: 622592 target oid smithi10548823-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 322560 snap -1 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:552: finishing set_chunk to oid 3 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1486848 length: 424448 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:554: finishing set_chunk to oid 5 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 803840 length: 974848 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:552: done (15 left) 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:554: done (14 left) 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:558 retry set_chunk ! 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 14 2024-04-20T02:50:43.081 INFO:tasks.rados.rados.0.smithi105.stdout:557: finishing set_chunk to oid 8 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 118272 length: 1825792 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:557: done (13 left) 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 13 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:553: finishing set_chunk to oid 4 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 914944 length: 1053184 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:555: finishing set_chunk to oid 6 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 832512 length: 317952 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:553: done (12 left) 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:555: done (11 left) 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 11 2024-04-20T02:50:43.082 INFO:tasks.rados.rados.0.smithi105.stdout:556: finishing set_chunk to oid 7 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 746496 length: 498688 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:556: done (10 left) 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 10 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:558: finishing set_chunk to oid 9 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 322560 length: 622592 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:558: done (9 left) 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 9 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:516: finishing set_chunk to oid 17 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:509: finishing set_chunk to oid 10 2024-04-20T02:50:43.083 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:509: done (8 left) 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:516: done (7 left) 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 7 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:539: finishing set_chunk to oid 40 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 21 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:525: finishing set_chunk to oid 26 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 15 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:525: done (6 left) 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:539: done (5 left) 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 5 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:515: finishing set_chunk to oid 16 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 9 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:526: finishing set_chunk to oid 27 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:43.084 INFO:tasks.rados.rados.0.smithi105.stdout:515: done (4 left) 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:526: done (3 left) 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 3 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:531: finishing set_chunk to oid 32 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:531: done (2 left) 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 2 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:550: finishing set_chunk to oid 1 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:550: done (1 left) 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 1 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:529: finishing set_chunk to oid 30 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 3 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:529: done (0 left) 2024-04-20T02:50:43.085 INFO:tasks.rados.rados.0.smithi105.stdout:559: set_chunk oid 10 target oid 10 low_tier 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:559: set_chunk oid 10 offset: 352768 length: 1298432 target oid smithi10548823-10 low_tier offset: 352768 snap -1 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:560: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:560: set_chunk oid 11 offset: 105472 length: 1453568 target oid smithi10548823-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 105472 snap -1 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:561: set_chunk oid 12 target oid 12 low_tier 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:561: set_chunk oid 12 offset: 216576 length: 780800 target oid smithi10548823-12 low_tier offset: 216576 snap -1 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:562: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:562: set_chunk oid 13 offset: 274944 length: 1643008 target oid smithi10548823-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 274944 snap -1 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:563: set_chunk oid 14 target oid 14 low_tier 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:563: set_chunk oid 14 offset: 253440 length: 287744 target oid smithi10548823-14 low_tier offset: 253440 snap -1 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:564: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:564: set_chunk oid 15 offset: 41984 length: 1188352 target oid smithi10548823-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 41984 snap -1 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:565: set_chunk oid 16 target oid 16 low_tier 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:565: set_chunk oid 16 offset: 60416 length: 186368 target oid smithi10548823-16 low_tier offset: 60416 snap -1 2024-04-20T02:50:43.086 INFO:tasks.rados.rados.0.smithi105.stdout:566: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:566: set_chunk oid 17 offset: 1484800 length: 393216 target oid smithi10548823-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1484800 snap -1 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:567: set_chunk oid 18 target oid 18 low_tier 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:567: set_chunk oid 18 offset: 1412096 length: 530432 target oid smithi10548823-18 low_tier offset: 1412096 snap -1 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:568: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:568: set_chunk oid 19 offset: 841728 length: 262144 target oid smithi10548823-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 841728 snap -1 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:569: set_chunk oid 20 target oid 20 low_tier 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:569: set_chunk oid 20 offset: 752128 length: 328704 target oid smithi10548823-20 low_tier offset: 752128 snap -1 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:570: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:570: set_chunk oid 21 offset: 171520 length: 276992 target oid smithi10548823-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 171520 snap -1 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:571: set_chunk oid 22 target oid 22 low_tier 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:571: set_chunk oid 22 offset: 1598976 length: 586752 target oid smithi10548823-22 low_tier offset: 1598976 snap -1 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:572: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:572: set_chunk oid 23 offset: 264704 length: 1536000 target oid smithi10548823-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 264704 snap -1 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:573: set_chunk oid 24 target oid 24 low_tier 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:573: set_chunk oid 24 offset: 1794560 length: 230400 target oid smithi10548823-24 low_tier offset: 1794560 snap -1 2024-04-20T02:50:43.087 INFO:tasks.rados.rados.0.smithi105.stdout:574: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:574: set_chunk oid 25 offset: 201216 length: 1719296 target oid smithi10548823-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 201216 snap -1 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:560: finishing set_chunk to oid 11 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 105472 length: 1453568 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:560: done (15 left) 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:575: set_chunk oid 26 target oid 26 low_tier 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:575: set_chunk oid 26 offset: 716800 length: 177664 target oid smithi10548823-26 low_tier offset: 716800 snap -1 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:559: finishing set_chunk to oid 10 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 352768 length: 1298432 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:562: finishing set_chunk to oid 13 2024-04-20T02:50:43.088 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 274944 length: 1643008 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:567: finishing set_chunk to oid 18 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1412096 length: 530432 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:564: finishing set_chunk to oid 15 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 41984 length: 1188352 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:561: finishing set_chunk to oid 12 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 216576 length: 780800 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:563: finishing set_chunk to oid 14 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 253440 length: 287744 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:565: finishing set_chunk to oid 16 2024-04-20T02:50:43.089 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 60416 length: 186368 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:571: finishing set_chunk to oid 22 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1598976 length: 586752 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:569: finishing set_chunk to oid 20 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 752128 length: 328704 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:570: finishing set_chunk to oid 21 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 171520 length: 276992 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:566: finishing set_chunk to oid 17 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1484800 length: 393216 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:568: finishing set_chunk to oid 19 2024-04-20T02:50:43.090 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 841728 length: 262144 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:572: finishing set_chunk to oid 23 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 264704 length: 1536000 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:574: finishing set_chunk to oid 25 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 201216 length: 1719296 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:575: finishing set_chunk to oid 26 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 716800 length: 177664 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:559: done (15 left) 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:561: done (14 left) 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:562: done (13 left) 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:563: done (12 left) 2024-04-20T02:50:43.091 INFO:tasks.rados.rados.0.smithi105.stdout:564: done (11 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:565: done (10 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:566: done (9 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:567: done (8 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:568: done (7 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:569: done (6 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:570: done (5 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:571: done (4 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:572: done (3 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:574: done (2 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:575: done (1 left) 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:576: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:576: set_chunk oid 27 offset: 387584 length: 457216 target oid smithi10548823-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 387584 snap -1 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:577: set_chunk oid 28 target oid 28 low_tier 2024-04-20T02:50:43.092 INFO:tasks.rados.rados.0.smithi105.stdout:577: set_chunk oid 28 offset: 74240 length: 1582080 target oid smithi10548823-28 low_tier offset: 74240 snap -1 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:578: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:578: set_chunk oid 29 offset: 277504 length: 1316352 target oid smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 277504 snap -1 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:579: set_chunk oid 30 target oid 30 low_tier 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:579: set_chunk oid 30 offset: 784384 length: 1656832 target oid smithi10548823-30 low_tier offset: 784384 snap -1 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:580: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:580: set_chunk oid 31 offset: 1466368 length: 1009664 target oid smithi10548823-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1466368 snap -1 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:581: set_chunk oid 32 target oid 32 low_tier 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:581: set_chunk oid 32 offset: 773632 length: 389120 target oid smithi10548823-32 low_tier offset: 773632 snap -1 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:582: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:582: set_chunk oid 33 offset: 115712 length: 1225728 target oid smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 115712 snap -1 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:583: set_chunk oid 34 target oid 34 low_tier 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:583: set_chunk oid 34 offset: 34816 length: 1130496 target oid smithi10548823-34 low_tier offset: 34816 snap -1 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:584: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.093 INFO:tasks.rados.rados.0.smithi105.stdout:584: set_chunk oid 35 offset: 1289216 length: 169984 target oid smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1289216 snap -1 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:585: set_chunk oid 36 target oid 36 low_tier 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:585: set_chunk oid 36 offset: 76288 length: 498688 target oid smithi10548823-36 low_tier offset: 76288 snap -1 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:586: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:586: set_chunk oid 37 offset: 172544 length: 1944064 target oid smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 172544 snap -1 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:587: set_chunk oid 38 target oid 38 low_tier 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:587: set_chunk oid 38 offset: 643072 length: 445952 target oid smithi10548823-38 low_tier offset: 643072 snap -1 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:588: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:588: set_chunk oid 39 offset: 2244608 length: 152064 target oid smithi10548823-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2244608 snap -1 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:589: set_chunk oid 40 target oid 40 low_tier 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:589: set_chunk oid 40 offset: 354304 length: 825856 target oid smithi10548823-40 low_tier offset: 354304 snap -1 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:590: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout:590: set_chunk oid 41 offset: 527360 length: 272384 target oid smithi10548823-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 527360 snap -1 2024-04-20T02:50:43.094 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:577: finishing set_chunk to oid 28 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 74240 length: 1582080 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:580: finishing set_chunk to oid 31 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1466368 length: 1009664 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:582: finishing set_chunk to oid 33 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 115712 length: 1225728 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:581: finishing set_chunk to oid 32 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 773632 length: 389120 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:584: finishing set_chunk to oid 35 2024-04-20T02:50:43.095 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1289216 length: 169984 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:578: finishing set_chunk to oid 29 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 277504 length: 1316352 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:579: finishing set_chunk to oid 30 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 784384 length: 1656832 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:577: done (15 left) 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:578: done (14 left) 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:579: done (13 left) 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:580: done (12 left) 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:581: done (11 left) 2024-04-20T02:50:43.096 INFO:tasks.rados.rados.0.smithi105.stdout:582: done (10 left) 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:584: done (9 left) 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:591: set_chunk oid 42 target oid 42 low_tier 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:591: set_chunk oid 42 offset: 166400 length: 1597440 target oid smithi10548823-42 low_tier offset: 166400 snap -1 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:592: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:592: set_chunk oid 43 offset: 924672 length: 947712 target oid smithi10548823-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 924672 snap -1 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:593: set_chunk oid 44 target oid 44 low_tier 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:593: set_chunk oid 44 offset: 743424 length: 962560 target oid smithi10548823-44 low_tier offset: 743424 snap -1 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:594: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:594: set_chunk oid 45 offset: 709120 length: 335872 target oid smithi10548823-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 709120 snap -1 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:595: set_chunk oid 46 target oid 46 low_tier 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:595: set_chunk oid 46 offset: 249344 length: 1987584 target oid smithi10548823-46 low_tier offset: 249344 snap -1 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:596: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:596: set_chunk oid 47 offset: 289280 length: 968192 target oid smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 289280 snap -1 2024-04-20T02:50:43.097 INFO:tasks.rados.rados.0.smithi105.stdout:597: set_chunk oid 48 target oid 48 low_tier 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:597: set_chunk oid 48 offset: 1690624 length: 346624 target oid smithi10548823-48 low_tier offset: 1690624 snap -1 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:586: finishing set_chunk to oid 37 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 172544 length: 1944064 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:586: done (15 left) 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:598: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:598: set_chunk oid 49 offset: 331264 length: 1412608 target oid smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 331264 snap -1 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:585: finishing set_chunk to oid 36 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 76288 length: 498688 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:576: finishing set_chunk to oid 27 2024-04-20T02:50:43.098 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 387584 length: 457216 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:589: finishing set_chunk to oid 40 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 354304 length: 825856 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:587: finishing set_chunk to oid 38 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 643072 length: 445952 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:591: finishing set_chunk to oid 42 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 166400 length: 1597440 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:593: finishing set_chunk to oid 44 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 743424 length: 962560 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:43.099 INFO:tasks.rados.rados.0.smithi105.stdout:594: finishing set_chunk to oid 45 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 709120 length: 335872 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:595: finishing set_chunk to oid 46 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 249344 length: 1987584 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:583: finishing set_chunk to oid 34 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 34816 length: 1130496 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:592: finishing set_chunk to oid 43 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 924672 length: 947712 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:576: done (15 left) 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:583: done (14 left) 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:585: done (13 left) 2024-04-20T02:50:43.100 INFO:tasks.rados.rados.0.smithi105.stdout:587: done (12 left) 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:589: done (11 left) 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:591: done (10 left) 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:592: done (9 left) 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:593: done (8 left) 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:594: done (7 left) 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:595: done (6 left) 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:599: set_chunk oid 50 target oid 50 low_tier 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:599: set_chunk oid 50 offset: 505856 length: 948736 target oid smithi10548823-50 low_tier offset: 505856 snap -1 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:600: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:600: set_chunk oid 1 offset: 281088 length: 1359360 target oid smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 281088 snap -1 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:601 oid_not_in_use 42 oid_redirect_not_in_use 0 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout:600 wait for manifest initialization 2024-04-20T02:50:43.101 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 8 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:596: finishing set_chunk to oid 47 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 289280 length: 968192 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:598: finishing set_chunk to oid 49 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 331264 length: 1412608 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:596: done (7 left) 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:598: done (6 left) 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 6 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:599: finishing set_chunk to oid 50 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 505856 length: 948736 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout:599: done (5 left) 2024-04-20T02:50:43.102 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 5 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:590: finishing set_chunk to oid 41 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 527360 length: 272384 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:597: finishing set_chunk to oid 48 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1690624 length: 346624 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:590: done (4 left) 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:597: done (3 left) 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 3 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:600: finishing set_chunk to oid 1 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 281088 length: 1359360 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout:600: done (2 left) 2024-04-20T02:50:43.103 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 2 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout:588: finishing set_chunk to oid 39 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout:588: done (1 left) 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 1 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout:573: finishing set_chunk to oid 24 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout:573: done (0 left) 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout:601 oid_not_in_use 50 oid_redirect_not_in_use 0 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 51 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 52 low_tier 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 53 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 54 low_tier 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 55 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 56 low_tier 2024-04-20T02:50:43.104 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 57 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 58 low_tier 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 59 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 60 low_tier 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 61 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 62 low_tier 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 63 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 64 low_tier 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 65 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 66 low_tier 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 67 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 68 low_tier 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 69 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 70 low_tier 2024-04-20T02:50:43.105 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 71 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.106 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 72 low_tier 2024-04-20T02:50:43.106 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 73 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.106 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 74 low_tier 2024-04-20T02:50:43.106 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 75 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.110 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 76 low_tier 2024-04-20T02:50:43.110 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 77 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.110 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 78 low_tier 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 79 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 80 low_tier 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 81 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 82 low_tier 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 83 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 84 low_tier 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 85 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 86 low_tier 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 87 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 88 low_tier 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 89 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 90 low_tier 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 91 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 92 low_tier 2024-04-20T02:50:43.111 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 93 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 94 low_tier 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 95 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 96 low_tier 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 97 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 98 low_tier 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 99 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout: redirect_not_in_use: 100 low_tier 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout:601: write oid 17 current snap is 0 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout:601: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout:601: writing smithi10548823-17 from 440350 to 900918 tid 1 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout:601: writing smithi10548823-17 from 1309629 to 1852295 tid 2 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout:601: writing smithi10548823-17 from 2323973 to 2840350 tid 3 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout:602: read oid 34 snap -1 2024-04-20T02:50:43.112 INFO:tasks.rados.rados.0.smithi105.stdout:602: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-20T02:50:43.113 INFO:tasks.rados.rados.0.smithi105.stdout:603: set_chunk oid 38 target oid 2024-04-20T02:50:43.114 INFO:tasks.rados.rados.0.smithi105.stdout:603: set_chunk oid 38 offset: 756736 length: 877056 target oid e50d02f2c145ae1a9f2fa318827bbec40426bb1a9882dc00d819884fd779b8f0 offset: 0 snap -1 2024-04-20T02:50:43.114 INFO:tasks.rados.rados.0.smithi105.stdout:604: rollback oid 4 current snap is 0 2024-04-20T02:50:43.114 INFO:tasks.rados.rados.0.smithi105.stdout:604: done (3 left) 2024-04-20T02:50:43.115 INFO:tasks.rados.rados.0.smithi105.stdout:605: write oid 9 current snap is 0 2024-04-20T02:50:43.115 INFO:tasks.rados.rados.0.smithi105.stdout:605: seq_num 51 ranges {457157=535817,1768111=402677} 2024-04-20T02:50:43.115 INFO:tasks.rados.rados.0.smithi105.stdout:605: writing smithi10548823-9 from 457157 to 992974 tid 1 2024-04-20T02:50:43.115 INFO:tasks.rados.rados.0.smithi105.stdout:605: writing smithi10548823-9 from 1768111 to 2170788 tid 2 2024-04-20T02:50:43.115 INFO:tasks.rados.rados.0.smithi105.stdout:606: set_chunk oid 29 target oid 2024-04-20T02:50:43.115 INFO:tasks.rados.rados.0.smithi105.stdout:606: set_chunk oid 29 offset: 271872 length: 1790464 target oid 8624d7f564bd805c1a61722fca95b7a15762ed4340977beb652509e48f02d2d4 offset: 0 snap -1 2024-04-20T02:50:43.115 INFO:tasks.rados.rados.0.smithi105.stdout:607: set_chunk oid 3 target oid 2024-04-20T02:50:43.115 INFO:tasks.rados.rados.0.smithi105.stdout:607: set_chunk oid 3 offset: 104448 length: 607744 target oid 90354acb35b5ef00820b10fdcfad8e09e1d29ee2922913977156ee1f1aaea7ca offset: 0 snap -1 2024-04-20T02:50:43.115 INFO:tasks.rados.rados.0.smithi105.stdout:608: write oid 37 current snap is 0 2024-04-20T02:50:43.115 INFO:tasks.rados.rados.0.smithi105.stdout:602: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-20T02:50:43.116 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:43.116 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:43.117 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:50:43.117 INFO:tasks.rados.rados.0.smithi105.stdout:603: finishing set_chunk to oid 38 2024-04-20T02:50:43.117 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 756736 length: 877056 2024-04-20T02:50:43.117 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-20T02:50:43.117 INFO:tasks.rados.rados.0.smithi105.stdout:601: finishing write tid 1 to smithi10548823-17 2024-04-20T02:50:43.117 INFO:tasks.rados.rados.0.smithi105.stdout:608: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-04-20T02:50:43.117 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:43.114+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:43.119 INFO:tasks.rados.rados.0.smithi105.stdout:608: writing smithi10548823-37 from 473964 to 1085030 tid 1 2024-04-20T02:50:43.127 INFO:tasks.rados.rados.0.smithi105.stdout:608: writing smithi10548823-37 from 1826593 to 2372928 tid 2 2024-04-20T02:50:43.131 INFO:tasks.rados.rados.0.smithi105.stdout:608: writing smithi10548823-37 from 2778819 to 2873964 tid 3 2024-04-20T02:50:43.131 INFO:tasks.rados.rados.0.smithi105.stdout:602: done (6 left) 2024-04-20T02:50:43.131 INFO:tasks.rados.rados.0.smithi105.stdout:603: done (5 left) 2024-04-20T02:50:43.131 INFO:tasks.rados.rados.0.smithi105.stdout:609: set_chunk oid 11 target oid 2024-04-20T02:50:43.136 INFO:tasks.rados.rados.0.smithi105.stdout:609: set_chunk oid 11 offset: 1380864 length: 173056 target oid b410c03415268664ecc17d37831b3bc40a25a9bddda966328b92cf1c65d9e5da offset: 0 snap -1 2024-04-20T02:50:43.136 INFO:tasks.rados.rados.0.smithi105.stdout:610: delete oid 16 current snap is 0 2024-04-20T02:50:43.136 INFO:tasks.rados.rados.0.smithi105.stdout:601: finishing write tid 2 to smithi10548823-17 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:601: finishing write tid 3 to smithi10548823-17 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:601: finishing write tid 4 to smithi10548823-17 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:601: finishing write tid 5 to smithi10548823-17 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:601: oid 17 updating version 0 to 16 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:601: oid 17 updating version 16 to 17 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:601: oid 17 updating version 17 to 19 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:601: oid 17 version 19 is already newer than 18 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:601: left oid 17 (ObjNum 50 snap 0 seq_num 50) 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:605: finishing write tid 1 to smithi10548823-9 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:605: finishing write tid 2 to smithi10548823-9 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:605: finishing write tid 3 to smithi10548823-9 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:605: finishing write tid 4 to smithi10548823-9 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:605: oid 9 updating version 0 to 29 2024-04-20T02:50:43.137 INFO:tasks.rados.rados.0.smithi105.stdout:605: oid 9 updating version 29 to 30 2024-04-20T02:50:43.138 INFO:tasks.rados.rados.0.smithi105.stdout:605: oid 9 updating version 30 to 31 2024-04-20T02:50:43.138 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 31 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-20T02:50:43.138 INFO:tasks.rados.rados.0.smithi105.stdout:605: left oid 9 (ObjNum 51 snap 0 seq_num 51) 2024-04-20T02:50:43.138 INFO:tasks.rados.rados.0.smithi105.stdout:606: finishing set_chunk to oid 29 2024-04-20T02:50:43.138 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk returned error code -95 offset: 271872 length: 1790464 2024-04-20T02:50:43.138 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-20T02:50:43.138 INFO:tasks.rados.rados.0.smithi105.stdout:607: finishing set_chunk to oid 3 2024-04-20T02:50:43.138 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 3 set_chunk returned error code -95 offset: 104448 length: 607744 2024-04-20T02:50:43.138 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:50:43.141 INFO:tasks.rados.rados.0.smithi105.stdout:601: done (6 left) 2024-04-20T02:50:43.141 INFO:tasks.rados.rados.0.smithi105.stdout:605: done (5 left) 2024-04-20T02:50:43.141 INFO:tasks.rados.rados.0.smithi105.stdout:606: done (4 left) 2024-04-20T02:50:43.141 INFO:tasks.rados.rados.0.smithi105.stdout:607: done (3 left) 2024-04-20T02:50:43.141 INFO:tasks.rados.rados.0.smithi105.stdout:610: done (2 left) 2024-04-20T02:50:43.141 INFO:tasks.rados.rados.0.smithi105.stdout:611: tier_flush oid 5 2024-04-20T02:50:43.142 INFO:tasks.rados.rados.0.smithi105.stdout:611: tier_flush oid 5 snap -1 2024-04-20T02:50:43.142 INFO:tasks.rados.rados.0.smithi105.stdout:612: read oid 22 snap -1 2024-04-20T02:50:43.142 INFO:tasks.rados.rados.0.smithi105.stdout:612: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-20T02:50:43.142 INFO:tasks.rados.rados.0.smithi105.stdout:613: read oid 26 snap -1 2024-04-20T02:50:43.142 INFO:tasks.rados.rados.0.smithi105.stdout:613: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-20T02:50:43.142 INFO:tasks.rados.rados.0.smithi105.stdout:614: write oid 30 current snap is 0 2024-04-20T02:50:43.142 INFO:tasks.rados.rados.0.smithi105.stdout:614: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-04-20T02:50:43.146 INFO:tasks.rados.rados.0.smithi105.stdout:614: writing smithi10548823-30 from 507578 to 985495 tid 1 2024-04-20T02:50:43.153 INFO:tasks.rados.rados.0.smithi105.stdout:614: writing smithi10548823-30 from 1543557 to 2093561 tid 2 2024-04-20T02:50:43.163 INFO:tasks.rados.rados.0.smithi105.stdout:614: writing smithi10548823-30 from 2833665 to 2907578 tid 3 2024-04-20T02:50:43.163 INFO:tasks.rados.rados.0.smithi105.stdout:615: read oid 2 snap -1 2024-04-20T02:50:43.163 INFO:tasks.rados.rados.0.smithi105.stdout:615: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-20T02:50:43.163 INFO:tasks.rados.rados.0.smithi105.stdout:616: read oid 38 snap -1 2024-04-20T02:50:43.163 INFO:tasks.rados.rados.0.smithi105.stdout:616: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:50:43.163 INFO:tasks.rados.rados.0.smithi105.stdout:617: write (excl) oid 1 current snap is 0 2024-04-20T02:50:43.163 INFO:tasks.rados.rados.0.smithi105.stdout:617: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-04-20T02:50:43.167 INFO:tasks.rados.rados.0.smithi105.stdout:617: writing smithi10548823-1 from 524385 to 1077551 tid 1 2024-04-20T02:50:43.175 INFO:tasks.rados.rados.0.smithi105.stdout:617: writing smithi10548823-1 from 1602039 to 2012054 tid 2 2024-04-20T02:50:43.179 INFO:tasks.rados.rados.0.smithi105.stdout:617: writing smithi10548823-1 from 2577441 to 2924385 tid 3 2024-04-20T02:50:43.179 INFO:tasks.rados.rados.0.smithi105.stdout:618: tier_promote oid 44 2024-04-20T02:50:43.179 INFO:tasks.rados.rados.0.smithi105.stdout:619: delete oid 13 current snap is 0 2024-04-20T02:50:43.188 INFO:tasks.rados.rados.0.smithi105.stdout:612: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-20T02:50:43.214 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:43.215+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:43.268 INFO:tasks.rados.rados.0.smithi105.stdout:609: finishing set_chunk to oid 11 2024-04-20T02:50:43.268 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:43.268 INFO:tasks.rados.rados.0.smithi105.stdout:609: done (10 left) 2024-04-20T02:50:43.269 INFO:tasks.rados.rados.0.smithi105.stdout:612: done (9 left) 2024-04-20T02:50:43.269 INFO:tasks.rados.rados.0.smithi105.stdout:619: done (8 left) 2024-04-20T02:50:43.269 INFO:tasks.rados.rados.0.smithi105.stdout:620: read oid 16 snap -1 2024-04-20T02:50:43.269 INFO:tasks.rados.rados.0.smithi105.stdout:620: expect deleted 2024-04-20T02:50:43.269 INFO:tasks.rados.rados.0.smithi105.stdout:621: write oid 4 current snap is 0 2024-04-20T02:50:43.269 INFO:tasks.rados.rados.0.smithi105.stdout:613: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-20T02:50:43.282 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:43.282 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:43.293 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:50:43.303 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:50:43.315 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:43.315+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:43.352 INFO:tasks.rados.rados.0.smithi105.stdout:616: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:50:43.415 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:43.416+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:43.422 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:43.422 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:43.433 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:50:43.438 INFO:tasks.rados.rados.0.smithi105.stdout:608: finishing write tid 1 to smithi10548823-37 2024-04-20T02:50:43.438 INFO:tasks.rados.rados.0.smithi105.stdout:608: finishing write tid 2 to smithi10548823-37 2024-04-20T02:50:43.438 INFO:tasks.rados.rados.0.smithi105.stdout:621: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-04-20T02:50:43.445 INFO:tasks.rados.rados.0.smithi105.stdout:621: writing smithi10548823-4 from 557999 to 1261663 tid 1 2024-04-20T02:50:43.450 INFO:tasks.rados.rados.0.smithi105.stdout:621: writing smithi10548823-4 from 1719003 to 2132687 tid 2 2024-04-20T02:50:43.455 INFO:tasks.rados.rados.0.smithi105.stdout:621: writing smithi10548823-4 from 2632287 to 2957999 tid 3 2024-04-20T02:50:43.455 INFO:tasks.rados.rados.0.smithi105.stdout:613: done (9 left) 2024-04-20T02:50:43.455 INFO:tasks.rados.rados.0.smithi105.stdout:616: done (8 left) 2024-04-20T02:50:43.455 INFO:tasks.rados.rados.0.smithi105.stdout:622: read oid 35 snap -1 2024-04-20T02:50:43.455 INFO:tasks.rados.rados.0.smithi105.stdout:622: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-20T02:50:43.455 INFO:tasks.rados.rados.0.smithi105.stdout:623: read oid 36 snap -1 2024-04-20T02:50:43.455 INFO:tasks.rados.rados.0.smithi105.stdout:623: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-20T02:50:43.455 INFO:tasks.rados.rados.0.smithi105.stdout:624: write oid 6 current snap is 0 2024-04-20T02:50:43.455 INFO:tasks.rados.rados.0.smithi105.stdout:624: seq_num 58 ranges {574806=778913,2061132=557342} 2024-04-20T02:50:43.462 INFO:tasks.rados.rados.0.smithi105.stdout:624: writing smithi10548823-6 from 574806 to 1353719 tid 1 2024-04-20T02:50:43.469 INFO:tasks.rados.rados.0.smithi105.stdout:624: writing smithi10548823-6 from 2061132 to 2618474 tid 2 2024-04-20T02:50:43.470 INFO:tasks.rados.rados.0.smithi105.stdout:625: write (excl) oid 39 current snap is 0 2024-04-20T02:50:43.470 INFO:tasks.rados.rados.0.smithi105.stdout:608: finishing write tid 3 to smithi10548823-37 2024-04-20T02:50:43.470 INFO:tasks.rados.rados.0.smithi105.stdout:608: finishing write tid 4 to smithi10548823-37 2024-04-20T02:50:43.470 INFO:tasks.rados.rados.0.smithi105.stdout:625: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-04-20T02:50:43.476 INFO:tasks.rados.rados.0.smithi105.stdout:625: writing smithi10548823-39 from 591613 to 1045775 tid 1 2024-04-20T02:50:43.481 INFO:tasks.rados.rados.0.smithi105.stdout:625: writing smithi10548823-39 from 1719614 to 2136967 tid 2 2024-04-20T02:50:43.487 INFO:tasks.rados.rados.0.smithi105.stdout:625: writing smithi10548823-39 from 2570780 to 2991613 tid 3 2024-04-20T02:50:43.487 INFO:tasks.rados.rados.0.smithi105.stdout:626: write oid 9 current snap is 0 2024-04-20T02:50:43.487 INFO:tasks.rados.rados.0.smithi105.stdout:608: finishing write tid 5 to smithi10548823-37 2024-04-20T02:50:43.487 INFO:tasks.rados.rados.0.smithi105.stdout:608: oid 37 updating version 0 to 16 2024-04-20T02:50:43.487 INFO:tasks.rados.rados.0.smithi105.stdout:608: oid 37 updating version 16 to 18 2024-04-20T02:50:43.487 INFO:tasks.rados.rados.0.smithi105.stdout:608: oid 37 version 18 is already newer than 17 2024-04-20T02:50:43.487 INFO:tasks.rados.rados.0.smithi105.stdout:608: oid 37 version 18 is already newer than 15 2024-04-20T02:50:43.487 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 18 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-04-20T02:50:43.487 INFO:tasks.rados.rados.0.smithi105.stdout:608: left oid 37 (ObjNum 52 snap 0 seq_num 52) 2024-04-20T02:50:43.487 INFO:tasks.rados.rados.0.smithi105.stdout:615: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-20T02:50:43.517 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:43.517+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:43.561 INFO:tasks.rados.rados.0.smithi105.stdout:614: finishing write tid 1 to smithi10548823-30 2024-04-20T02:50:43.561 INFO:tasks.rados.rados.0.smithi105.stdout:614: finishing write tid 2 to smithi10548823-30 2024-04-20T02:50:43.561 INFO:tasks.rados.rados.0.smithi105.stdout:614: finishing write tid 3 to smithi10548823-30 2024-04-20T02:50:43.561 INFO:tasks.rados.rados.0.smithi105.stdout:614: finishing write tid 4 to smithi10548823-30 2024-04-20T02:50:43.561 INFO:tasks.rados.rados.0.smithi105.stdout:626: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-04-20T02:50:43.567 INFO:tasks.rados.rados.0.smithi105.stdout:626: writing smithi10548823-9 from 608420 to 1137831 tid 1 2024-04-20T02:50:43.576 INFO:tasks.rados.rados.0.smithi105.stdout:626: writing smithi10548823-9 from 1778096 to 2339107 tid 2 2024-04-20T02:50:43.579 INFO:tasks.rados.rados.0.smithi105.stdout:626: writing smithi10548823-9 from 2881850 to 3008420 tid 3 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:608: done (12 left) 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:615: done (11 left) 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:627: write (excl) oid 13 current snap is 0 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:614: finishing write tid 5 to smithi10548823-30 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:614: oid 30 updating version 0 to 15 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:614: oid 30 version 15 is already newer than 14 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:614: oid 30 version 15 is already newer than 13 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:614: oid 30 version 15 is already newer than 12 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 15 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:614: left oid 30 (ObjNum 54 snap 0 seq_num 54) 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:611: got (0) Success 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-20T02:50:43.580 INFO:tasks.rados.rados.0.smithi105.stdout:617: finishing write tid 1 to smithi10548823-1 2024-04-20T02:50:43.581 INFO:tasks.rados.rados.0.smithi105.stdout:627: seq_num 61 ranges {625227=721013,1952931=704669} 2024-04-20T02:50:43.587 INFO:tasks.rados.rados.0.smithi105.stdout:627: writing smithi10548823-13 from 625227 to 1346240 tid 1 2024-04-20T02:50:43.596 INFO:tasks.rados.rados.0.smithi105.stdout:627: writing smithi10548823-13 from 1952931 to 2657600 tid 2 2024-04-20T02:50:43.596 INFO:tasks.rados.rados.0.smithi105.stdout:611: done (11 left) 2024-04-20T02:50:43.596 INFO:tasks.rados.rados.0.smithi105.stdout:614: done (10 left) 2024-04-20T02:50:43.596 INFO:tasks.rados.rados.0.smithi105.stdout:628: delete oid 10 current snap is 0 2024-04-20T02:50:43.597 INFO:tasks.rados.rados.0.smithi105.stdout:618: got (0) Success 2024-04-20T02:50:43.597 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-20T02:50:43.597 INFO:tasks.rados.rados.0.smithi105.stdout:617: finishing write tid 2 to smithi10548823-1 2024-04-20T02:50:43.597 INFO:tasks.rados.rados.0.smithi105.stdout:617: finishing write tid 3 to smithi10548823-1 2024-04-20T02:50:43.597 INFO:tasks.rados.rados.0.smithi105.stdout:617: finishing write tid 4 to smithi10548823-1 2024-04-20T02:50:43.597 INFO:tasks.rados.rados.0.smithi105.stdout:617: finishing write tid 5 to smithi10548823-1 2024-04-20T02:50:43.597 INFO:tasks.rados.rados.0.smithi105.stdout:617: oid 1 updating version 0 to 35 2024-04-20T02:50:43.598 INFO:tasks.rados.rados.0.smithi105.stdout:617: oid 1 version 35 is already newer than 32 2024-04-20T02:50:43.598 INFO:tasks.rados.rados.0.smithi105.stdout:617: oid 1 version 35 is already newer than 34 2024-04-20T02:50:43.598 INFO:tasks.rados.rados.0.smithi105.stdout:617: oid 1 version 35 is already newer than 33 2024-04-20T02:50:43.598 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 35 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-04-20T02:50:43.598 INFO:tasks.rados.rados.0.smithi105.stdout:617: left oid 1 (ObjNum 55 snap 0 seq_num 55) 2024-04-20T02:50:43.598 INFO:tasks.rados.rados.0.smithi105.stdout:622: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-20T02:50:43.611 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:43.611 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:43.617 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:43.617+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:43.625 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:50:43.683 INFO:tasks.rados.rados.0.smithi105.stdout:623: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-20T02:50:43.718 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:43.719+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:43.755 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:43.756 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:43.765 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:50:43.777 INFO:tasks.rados.rados.0.smithi105.stdout:621: finishing write tid 1 to smithi10548823-4 2024-04-20T02:50:43.777 INFO:tasks.rados.rados.0.smithi105.stdout:621: finishing write tid 2 to smithi10548823-4 2024-04-20T02:50:43.777 INFO:tasks.rados.rados.0.smithi105.stdout:621: finishing write tid 3 to smithi10548823-4 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:621: finishing write tid 4 to smithi10548823-4 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:621: finishing write tid 5 to smithi10548823-4 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:621: oid 4 updating version 0 to 39 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:621: oid 4 version 39 is already newer than 37 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:621: oid 4 version 39 is already newer than 36 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:621: oid 4 version 39 is already newer than 38 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 39 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:621: left oid 4 (ObjNum 57 snap 0 seq_num 57) 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:624: finishing write tid 1 to smithi10548823-6 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:624: finishing write tid 2 to smithi10548823-6 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:624: finishing write tid 3 to smithi10548823-6 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:624: finishing write tid 4 to smithi10548823-6 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:624: oid 6 updating version 0 to 21 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:624: oid 6 updating version 21 to 23 2024-04-20T02:50:43.778 INFO:tasks.rados.rados.0.smithi105.stdout:624: oid 6 version 23 is already newer than 22 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 23 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:624: left oid 6 (ObjNum 58 snap 0 seq_num 58) 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:626: finishing write tid 1 to smithi10548823-9 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:617: done (10 left) 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:618: done (9 left) 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:620: done (8 left) 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:621: done (7 left) 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:622: done (6 left) 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:623: done (5 left) 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:624: done (4 left) 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:628: done (3 left) 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:629: write (excl) oid 10 current snap is 0 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:626: finishing write tid 2 to smithi10548823-9 2024-04-20T02:50:43.779 INFO:tasks.rados.rados.0.smithi105.stdout:626: finishing write tid 3 to smithi10548823-9 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:626: finishing write tid 4 to smithi10548823-9 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:626: finishing write tid 5 to smithi10548823-9 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:626: oid 9 updating version 0 to 41 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:626: oid 9 version 41 is already newer than 40 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:626: oid 9 updating version 41 to 42 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:626: oid 9 updating version 42 to 43 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 43 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:626: left oid 9 (ObjNum 60 snap 0 seq_num 60) 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:627: finishing write tid 1 to smithi10548823-13 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:627: finishing write tid 2 to smithi10548823-13 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:627: finishing write tid 3 to smithi10548823-13 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:627: finishing write tid 4 to smithi10548823-13 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:627: oid 13 updating version 0 to 46 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:627: oid 13 version 46 is already newer than 44 2024-04-20T02:50:43.780 INFO:tasks.rados.rados.0.smithi105.stdout:627: oid 13 version 46 is already newer than 45 2024-04-20T02:50:43.781 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 46 (ObjNum 61 snap 0 seq_num 61) dirty exists 2024-04-20T02:50:43.781 INFO:tasks.rados.rados.0.smithi105.stdout:627: left oid 13 (ObjNum 61 snap 0 seq_num 61) 2024-04-20T02:50:43.781 INFO:tasks.rados.rados.0.smithi105.stdout:625: finishing write tid 1 to smithi10548823-39 2024-04-20T02:50:43.781 INFO:tasks.rados.rados.0.smithi105.stdout:625: finishing write tid 2 to smithi10548823-39 2024-04-20T02:50:43.781 INFO:tasks.rados.rados.0.smithi105.stdout:625: finishing write tid 3 to smithi10548823-39 2024-04-20T02:50:43.781 INFO:tasks.rados.rados.0.smithi105.stdout:629: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-04-20T02:50:43.782 INFO:tasks.rados.rados.0.smithi105.stdout:629: writing smithi10548823-10 from 658841 to 1130352 tid 1 2024-04-20T02:50:43.789 INFO:tasks.rados.rados.0.smithi105.stdout:629: writing smithi10548823-10 from 1553542 to 2261880 tid 2 2024-04-20T02:50:43.793 INFO:tasks.rados.rados.0.smithi105.stdout:629: writing smithi10548823-10 from 2964119 to 3058841 tid 3 2024-04-20T02:50:43.794 INFO:tasks.rados.rados.0.smithi105.stdout:626: done (3 left) 2024-04-20T02:50:43.794 INFO:tasks.rados.rados.0.smithi105.stdout:627: done (2 left) 2024-04-20T02:50:43.794 INFO:tasks.rados.rados.0.smithi105.stdout:630: read oid 26 snap -1 2024-04-20T02:50:43.794 INFO:tasks.rados.rados.0.smithi105.stdout:630: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-20T02:50:43.794 INFO:tasks.rados.rados.0.smithi105.stdout:631: tier_evict oid 38 2024-04-20T02:50:43.794 INFO:tasks.rados.rados.0.smithi105.stdout:631: tier_evict oid 38 snap -1 2024-04-20T02:50:43.794 INFO:tasks.rados.rados.0.smithi105.stdout:625: finishing write tid 4 to smithi10548823-39 2024-04-20T02:50:43.794 INFO:tasks.rados.rados.0.smithi105.stdout:632: set_chunk oid 11 target oid 2024-04-20T02:50:43.819 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:43.820+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:43.825 INFO:teuthology.orchestra.run.smithi138.stdout: marking collection for removal 2024-04-20T02:50:43.825 INFO:teuthology.orchestra.run.smithi138.stdout:setting '_remove' omap key 2024-04-20T02:50:43.825 INFO:teuthology.orchestra.run.smithi138.stdout:finish_remove_pgs 4.2_head removing 4.2 2024-04-20T02:50:43.827 INFO:teuthology.orchestra.run.smithi138.stdout:Remove successful 2024-04-20T02:50:43.828 INFO:tasks.rados.rados.0.smithi105.stdout:632: set_chunk oid 11 offset: 714752 length: 697344 target oid f26cb2f3c6ae5d3ec1685f6097f6e96fe3fcbb5845f1f6fc0a09361ad5871afd offset: 0 snap -1 2024-04-20T02:50:43.828 INFO:tasks.rados.rados.0.smithi105.stdout:633: read oid 25 snap -1 2024-04-20T02:50:43.828 INFO:tasks.rados.rados.0.smithi105.stdout:633: expect (ObjNum 24 snap 0 seq_num 24) 2024-04-20T02:50:43.828 INFO:tasks.rados.rados.0.smithi105.stdout:634: write (excl) oid 30 current snap is 0 2024-04-20T02:50:43.829 INFO:tasks.rados.rados.0.smithi105.stdout:625: finishing write tid 5 to smithi10548823-39 2024-04-20T02:50:43.829 INFO:tasks.rados.rados.0.smithi105.stdout:625: oid 39 updating version 0 to 30 2024-04-20T02:50:43.829 INFO:tasks.rados.rados.0.smithi105.stdout:625: oid 39 version 30 is already newer than 28 2024-04-20T02:50:43.829 INFO:tasks.rados.rados.0.smithi105.stdout:625: oid 39 updating version 30 to 31 2024-04-20T02:50:43.829 INFO:tasks.rados.rados.0.smithi105.stdout:625: oid 39 version 31 is already newer than 29 2024-04-20T02:50:43.829 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 31 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-04-20T02:50:43.829 INFO:tasks.rados.rados.0.smithi105.stdout:625: left oid 39 (ObjNum 59 snap 0 seq_num 59) 2024-04-20T02:50:43.829 INFO:tasks.rados.rados.0.smithi105.stdout:634: seq_num 64 ranges {675648=546760,2012024=568349} 2024-04-20T02:50:43.834 INFO:tasks.rados.rados.0.smithi105.stdout:634: writing smithi10548823-30 from 675648 to 1222408 tid 1 2024-04-20T02:50:43.842 INFO:tasks.rados.rados.0.smithi105.stdout:634: writing smithi10548823-30 from 2012024 to 2580373 tid 2 2024-04-20T02:50:43.842 INFO:tasks.rados.rados.0.smithi105.stdout:625: done (6 left) 2024-04-20T02:50:43.842 INFO:tasks.rados.rados.0.smithi105.stdout:635: write (excl) oid 12 current snap is 0 2024-04-20T02:50:43.842 INFO:tasks.rados.rados.0.smithi105.stdout:631: got (22) Invalid argument 2024-04-20T02:50:43.842 INFO:tasks.rados.rados.0.smithi105.stdout:635: seq_num 65 ranges {692455=622009,2070506=712007} 2024-04-20T02:50:43.850 INFO:tasks.rados.rados.0.smithi105.stdout:635: writing smithi10548823-12 from 692455 to 1314464 tid 1 2024-04-20T02:50:43.860 INFO:tasks.rados.rados.0.smithi105.stdout:635: writing smithi10548823-12 from 2070506 to 2782513 tid 2 2024-04-20T02:50:43.860 INFO:tasks.rados.rados.0.smithi105.stdout:631: done (6 left) 2024-04-20T02:50:43.860 INFO:tasks.rados.rados.0.smithi105.stdout:636: read oid 13 snap -1 2024-04-20T02:50:43.860 INFO:tasks.rados.rados.0.smithi105.stdout:636: expect (ObjNum 61 snap 0 seq_num 61) 2024-04-20T02:50:43.860 INFO:tasks.rados.rados.0.smithi105.stdout:637: write oid 44 current snap is 0 2024-04-20T02:50:43.860 INFO:tasks.rados.rados.0.smithi105.stdout:637: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-04-20T02:50:43.868 INFO:tasks.rados.rados.0.smithi105.stdout:637: writing smithi10548823-44 from 709262 to 1406520 tid 1 2024-04-20T02:50:43.874 INFO:tasks.rados.rados.0.smithi105.stdout:637: writing smithi10548823-44 from 2012635 to 2584653 tid 2 2024-04-20T02:50:43.878 INFO:tasks.rados.rados.0.smithi105.stdout:637: writing smithi10548823-44 from 3046388 to 3109262 tid 3 2024-04-20T02:50:43.878 INFO:tasks.rados.rados.0.smithi105.stdout:638: write oid 23 current snap is 0 2024-04-20T02:50:43.878 INFO:tasks.rados.rados.0.smithi105.stdout:630: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-20T02:50:43.918 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:43.918 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:43.920 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:43.921+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:43.929 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:50:43.962 INFO:tasks.rados.rados.0.smithi105.stdout:629: finishing write tid 1 to smithi10548823-10 2024-04-20T02:50:43.962 INFO:tasks.rados.rados.0.smithi105.stdout:629: finishing write tid 2 to smithi10548823-10 2024-04-20T02:50:43.962 INFO:tasks.rados.rados.0.smithi105.stdout:638: seq_num 67 ranges {726069=772507,2071117=715676} 2024-04-20T02:50:43.970 INFO:tasks.rados.rados.0.smithi105.stdout:638: writing smithi10548823-23 from 726069 to 1498576 tid 1 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:638: writing smithi10548823-23 from 2071117 to 2786793 tid 2 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:630: done (8 left) 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:639: read oid 14 snap -1 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:639: expect (ObjNum 13 snap 0 seq_num 13) 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:640: delete oid 19 current snap is 0 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:629: finishing write tid 3 to smithi10548823-10 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:629: finishing write tid 4 to smithi10548823-10 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:629: finishing write tid 5 to smithi10548823-10 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:629: oid 10 updating version 0 to 33 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:629: oid 10 updating version 33 to 34 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:629: oid 10 version 34 is already newer than 32 2024-04-20T02:50:43.979 INFO:tasks.rados.rados.0.smithi105.stdout:629: oid 10 version 34 is already newer than 31 2024-04-20T02:50:43.980 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 34 (ObjNum 63 snap 0 seq_num 63) dirty exists 2024-04-20T02:50:43.980 INFO:tasks.rados.rados.0.smithi105.stdout:629: left oid 10 (ObjNum 63 snap 0 seq_num 63) 2024-04-20T02:50:43.980 INFO:tasks.rados.rados.0.smithi105.stdout:632: finishing set_chunk to oid 11 2024-04-20T02:50:43.980 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 11 set_chunk returned error code -95 offset: 714752 length: 697344 2024-04-20T02:50:43.980 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-20T02:50:43.980 INFO:tasks.rados.rados.0.smithi105.stdout:633: expect (ObjNum 24 snap 0 seq_num 24) 2024-04-20T02:50:44.021 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:44.021+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:44.059 INFO:tasks.rados.rados.0.smithi105.stdout:634: finishing write tid 1 to smithi10548823-30 2024-04-20T02:50:44.059 INFO:tasks.rados.rados.0.smithi105.stdout:634: finishing write tid 2 to smithi10548823-30 2024-04-20T02:50:44.059 INFO:tasks.rados.rados.0.smithi105.stdout:634: finishing write tid 3 to smithi10548823-30 2024-04-20T02:50:44.060 INFO:tasks.rados.rados.0.smithi105.stdout:634: finishing write tid 4 to smithi10548823-30 2024-04-20T02:50:44.060 INFO:tasks.rados.rados.0.smithi105.stdout:634: oid 30 updating version 0 to 16 2024-04-20T02:50:44.060 INFO:tasks.rados.rados.0.smithi105.stdout:634: oid 30 updating version 16 to 18 2024-04-20T02:50:44.060 INFO:tasks.rados.rados.0.smithi105.stdout:634: oid 30 version 18 is already newer than 17 2024-04-20T02:50:44.060 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 18 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-04-20T02:50:44.060 INFO:tasks.rados.rados.0.smithi105.stdout:634: left oid 30 (ObjNum 64 snap 0 seq_num 64) 2024-04-20T02:50:44.060 INFO:tasks.rados.rados.0.smithi105.stdout:636: expect (ObjNum 61 snap 0 seq_num 61) 2024-04-20T02:50:44.098 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:44.098 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:44.108 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:50:44.122 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:44.123+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:44.157 INFO:tasks.rados.rados.0.smithi105.stdout:635: finishing write tid 1 to smithi10548823-12 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:637: finishing write tid 1 to smithi10548823-44 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:635: finishing write tid 2 to smithi10548823-12 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:637: finishing write tid 2 to smithi10548823-44 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:635: finishing write tid 3 to smithi10548823-12 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:635: finishing write tid 4 to smithi10548823-12 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:635: oid 12 updating version 0 to 7 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:635: oid 12 version 7 is already newer than 6 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:635: oid 12 updating version 7 to 8 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 8 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:635: left oid 12 (ObjNum 65 snap 0 seq_num 65) 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:637: finishing write tid 3 to smithi10548823-44 2024-04-20T02:50:44.158 INFO:tasks.rados.rados.0.smithi105.stdout:629: done (9 left) 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:632: done (8 left) 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:633: done (7 left) 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:634: done (6 left) 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:635: done (5 left) 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:636: done (4 left) 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:640: done (3 left) 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:641: delete oid 20 current snap is 0 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:637: finishing write tid 4 to smithi10548823-44 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:637: finishing write tid 5 to smithi10548823-44 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:637: oid 44 updating version 0 to 14 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:637: oid 44 updating version 14 to 15 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:637: oid 44 updating version 15 to 16 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:637: oid 44 updating version 16 to 17 2024-04-20T02:50:44.159 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 17 (ObjNum 66 snap 0 seq_num 66) dirty exists 2024-04-20T02:50:44.160 INFO:tasks.rados.rados.0.smithi105.stdout:637: left oid 44 (ObjNum 66 snap 0 seq_num 66) 2024-04-20T02:50:44.160 INFO:tasks.rados.rados.0.smithi105.stdout:639: expect (ObjNum 13 snap 0 seq_num 13) 2024-04-20T02:50:44.223 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:44.224+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:44.231 INFO:tasks.rados.rados.0.smithi105.stdout:638: finishing write tid 1 to smithi10548823-23 2024-04-20T02:50:44.231 INFO:tasks.rados.rados.0.smithi105.stdout:638: finishing write tid 2 to smithi10548823-23 2024-04-20T02:50:44.231 INFO:tasks.rados.rados.0.smithi105.stdout:638: finishing write tid 3 to smithi10548823-23 2024-04-20T02:50:44.231 INFO:tasks.rados.rados.0.smithi105.stdout:638: finishing write tid 4 to smithi10548823-23 2024-04-20T02:50:44.231 INFO:tasks.rados.rados.0.smithi105.stdout:638: oid 23 updating version 0 to 21 2024-04-20T02:50:44.231 INFO:tasks.rados.rados.0.smithi105.stdout:638: oid 23 version 21 is already newer than 20 2024-04-20T02:50:44.231 INFO:tasks.rados.rados.0.smithi105.stdout:638: oid 23 version 21 is already newer than 19 2024-04-20T02:50:44.231 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 21 (ObjNum 67 snap 0 seq_num 67) dirty exists 2024-04-20T02:50:44.231 INFO:tasks.rados.rados.0.smithi105.stdout:638: left oid 23 (ObjNum 67 snap 0 seq_num 67) 2024-04-20T02:50:44.232 INFO:tasks.rados.rados.0.smithi105.stdout:637: done (3 left) 2024-04-20T02:50:44.232 INFO:tasks.rados.rados.0.smithi105.stdout:638: done (2 left) 2024-04-20T02:50:44.232 INFO:tasks.rados.rados.0.smithi105.stdout:639: done (1 left) 2024-04-20T02:50:44.232 INFO:tasks.rados.rados.0.smithi105.stdout:641: done (0 left) 2024-04-20T02:50:44.232 INFO:tasks.rados.rados.0.smithi105.stdout:642: read oid 33 snap -1 2024-04-20T02:50:44.232 INFO:tasks.rados.rados.0.smithi105.stdout:642: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-20T02:50:44.232 INFO:tasks.rados.rados.0.smithi105.stdout:643: set_chunk oid 41 target oid 2024-04-20T02:50:44.237 INFO:tasks.rados.rados.0.smithi105.stdout:643: set_chunk oid 41 offset: 1282048 length: 175616 target oid c8d865012412e7a1158eefa55e138e9e72ae14b6dd5812627d8da231df16006f offset: 0 snap -1 2024-04-20T02:50:44.237 INFO:tasks.rados.rados.0.smithi105.stdout:644: read oid 24 snap -1 2024-04-20T02:50:44.238 INFO:tasks.rados.rados.0.smithi105.stdout:644: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-20T02:50:44.238 INFO:tasks.rados.rados.0.smithi105.stdout:645: read oid 22 snap -1 2024-04-20T02:50:44.238 INFO:tasks.rados.rados.0.smithi105.stdout:645: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-20T02:50:44.238 INFO:tasks.rados.rados.0.smithi105.stdout:646: read oid 49 snap -1 2024-04-20T02:50:44.238 INFO:tasks.rados.rados.0.smithi105.stdout:646: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-20T02:50:44.238 INFO:tasks.rados.rados.0.smithi105.stdout:647: tier_flush oid 27 2024-04-20T02:50:44.238 INFO:tasks.rados.rados.0.smithi105.stdout:647: tier_flush oid 27 snap -1 2024-04-20T02:50:44.238 INFO:tasks.rados.rados.0.smithi105.stdout:648: read oid 28 snap -1 2024-04-20T02:50:44.238 INFO:tasks.rados.rados.0.smithi105.stdout:648: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-20T02:50:44.238 INFO:tasks.rados.rados.0.smithi105.stdout:649: set_chunk oid 44 target oid 2024-04-20T02:50:44.239 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:44.239 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:44.253 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:50:44.287 INFO:tasks.rados.rados.0.smithi105.stdout:649: set_chunk oid 44 offset: 204288 length: 903680 target oid 65a9b671d26c663054dc1fa0ed1026723b6fd1f1301037997ae2700e393a5dd4 offset: 0 snap -1 2024-04-20T02:50:44.287 INFO:tasks.rados.rados.0.smithi105.stdout:650: write oid 25 current snap is 0 2024-04-20T02:50:44.287 INFO:tasks.rados.rados.0.smithi105.stdout:643: finishing set_chunk to oid 41 2024-04-20T02:50:44.287 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 41 set_chunk returned error code -95 offset: 1282048 length: 175616 2024-04-20T02:50:44.287 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-20T02:50:44.287 INFO:tasks.rados.rados.0.smithi105.stdout:650: seq_num 70 ranges {776490=714607,2246563=579356} 2024-04-20T02:50:44.295 INFO:tasks.rados.rados.0.smithi105.stdout:650: writing smithi10548823-25 from 776490 to 1491097 tid 1 2024-04-20T02:50:44.302 INFO:tasks.rados.rados.0.smithi105.stdout:650: writing smithi10548823-25 from 2246563 to 2825919 tid 2 2024-04-20T02:50:44.302 INFO:tasks.rados.rados.0.smithi105.stdout:643: done (8 left) 2024-04-20T02:50:44.302 INFO:tasks.rados.rados.0.smithi105.stdout:651: delete oid 15 current snap is 0 2024-04-20T02:50:44.302 INFO:tasks.rados.rados.0.smithi105.stdout:645: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-20T02:50:44.339 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:44.340+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:44.350 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.2.5 2024-04-20T02:50:44.385 INFO:tasks.rados.rados.0.smithi105.stdout:642: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-20T02:50:44.431 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:44.432 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:44.432 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:44.432+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:44.443 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:50:44.469 INFO:tasks.rados.rados.0.smithi105.stdout:644: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-20T02:50:44.527 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:44.528+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:44.547 INFO:tasks.rados.rados.0.smithi105.stdout:646: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-20T02:50:44.612 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:44.612 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:44.622 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:50:44.626 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:44.627+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:44.637 INFO:tasks.rados.rados.0.smithi105.stdout:648: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:647: got (0) Success 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:649: finishing set_chunk to oid 44 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 17 (ObjNum 66 snap 0 seq_num 66) dirty exists 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:650: finishing write tid 1 to smithi10548823-25 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:650: finishing write tid 2 to smithi10548823-25 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:650: finishing write tid 3 to smithi10548823-25 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:650: finishing write tid 4 to smithi10548823-25 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:650: oid 25 updating version 0 to 28 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:650: oid 25 version 28 is already newer than 27 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:650: oid 25 version 28 is already newer than 26 2024-04-20T02:50:44.716 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 28 (ObjNum 70 snap 0 seq_num 70) dirty exists 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:650: left oid 25 (ObjNum 70 snap 0 seq_num 70) 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:642: done (8 left) 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:644: done (7 left) 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:645: done (6 left) 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:646: done (5 left) 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:647: done (4 left) 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:648: done (3 left) 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:649: done (2 left) 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:650: done (1 left) 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:651: done (0 left) 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:652: tier_flush oid 4 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:652: tier_flush oid 4 snap -1 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:653: tier_flush oid 8 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:653: tier_flush oid 8 snap -1 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:654: read oid 12 snap -1 2024-04-20T02:50:44.717 INFO:tasks.rados.rados.0.smithi105.stdout:654: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-20T02:50:44.718 INFO:tasks.rados.rados.0.smithi105.stdout:655: snap_create 2024-04-20T02:50:44.718 INFO:tasks.rados.rados.0.smithi105.stdout:654: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-20T02:50:44.731 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:44.731+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:44.751 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:50:44.819 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:44.819 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:44.828 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:44.828+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:44.830 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:50:44.870 INFO:tasks.rados.rados.0.smithi105.stdout:653: got (0) Success 2024-04-20T02:50:44.870 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-20T02:50:44.870 INFO:tasks.rados.rados.0.smithi105.stdout:652: got (0) Success 2024-04-20T02:50:44.871 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-04-20T02:50:44.902 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:50:44.921 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:50:44.929 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:44.929+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:44.940 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:50:44.959 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:50:44.963 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:44.963 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:44.963 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:50:44.963 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:44.974 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:50:44.979 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:50:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:50:45.022 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:50:45.022 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:50:45.023 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:50:45.023 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:50:45.023 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:50:45.023 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:50:45.023 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-20T02:50:45.023 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:50:45.023 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:50:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:50:45.025 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:50:45.067 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:45.067+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:45.109 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:50:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:50:45.109 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:45.119 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:50:45.160 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:45.160+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:45.233 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:45.233+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:45.289 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:45.289 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:50:45.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:50:45.289 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:50:45.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.289 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:1 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'1, mlcod=0'0)", 2024-04-20T02:50:45.289 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.924154+0000", 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.364581019999999, 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003869146, 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924154+0000", 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924154+0000", 2024-04-20T02:50:45.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924153+0000", 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924159+0000", 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.161e-06 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924172+0000", 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2300000000000001e-05 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924182+0000", 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0332999999999999e-05 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924218+0000", 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5800000000000003e-05 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.924245+0000", 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6954e-05 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.927990+0000", 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003745253 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.928023+0000", 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2685000000000003e-05 2024-04-20T02:50:45.293 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:33 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'19, mlcod=16'14)", 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.963561+0000", 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.325173767999999, 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040083649999999998, 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963561+0000", 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963561+0000", 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963560+0000", 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963567+0000", 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3609999999999997e-06 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963576+0000", 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4669999999999993e-06 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963583+0000", 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3250000000000002e-06 2024-04-20T02:50:45.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963635+0000", 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1597000000000001e-05 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.963654+0000", 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9531999999999999e-05 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967545+0000", 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0038906840000000002 2024-04-20T02:50:45.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967569+0000", 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4108e-05 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:34 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'20, mlcod=16'15)", 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.964277+0000", 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.324457936000002, 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033064570000000001, 2024-04-20T02:50:45.298 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964277+0000", 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964277+0000", 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964276+0000", 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964282+0000", 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9800000000000003e-06 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964291+0000", 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9029999999999993e-06 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964296+0000", 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.101e-06 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964322+0000", 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5647e-05 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964336+0000", 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4452e-05 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967573+0000", 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003236591 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967583+0000", 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0429e-05 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:35 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'21, mlcod=16'15)", 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.964635+0000", 2024-04-20T02:50:45.302 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.324099775000001, 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037872499999999998, 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964635+0000", 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964635+0000", 2024-04-20T02:50:45.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964634+0000", 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964689+0000", 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4784999999999998e-05 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964696+0000", 2024-04-20T02:50:45.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9870000000000002e-06 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.964700+0000", 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.031e-06 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967084+0000", 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0023832760000000001 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967131+0000", 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7444999999999997e-05 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968377+0000", 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001246055 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968422+0000", 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5089000000000001e-05 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:36 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'22, mlcod=16'16)", 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.965393+0000", 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.323341766999999, 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0035673390000000001, 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965393+0000", 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965393+0000", 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965392+0000", 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965411+0000", 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8612000000000001e-05 2024-04-20T02:50:45.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965417+0000", 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.913e-06 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965421+0000", 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8630000000000001e-06 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967375+0000", 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019539560000000002 2024-04-20T02:50:45.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967389+0000", 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4496e-05 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968940+0000", 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015508939999999999 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968960+0000", 2024-04-20T02:50:45.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0037000000000001e-05 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:37 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'23, mlcod=16'16)", 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:19.965893+0000", 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.322841948000001, 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030820349999999999, 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.311 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965893+0000", 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965893+0000", 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965892+0000", 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965896+0000", 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.16e-06 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965904+0000", 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0879999999999997e-06 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.965907+0000", 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.574e-06 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967603+0000", 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001696078 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.967618+0000", 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4718e-05 2024-04-20T02:50:45.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968962+0000", 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001344095 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:19.968975+0000", 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734999999999999e-05 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:115 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'77, mlcod=16'66)", 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.141789+0000", 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.146945834, 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0029188780000000002, 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141789+0000", 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141789+0000", 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141789+0000", 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141861+0000", 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2187999999999994e-05 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141866+0000", 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4700000000000001e-06 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.141870+0000", 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1420000000000003e-06 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.142829+0000", 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00095890999999999999 2024-04-20T02:50:45.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.142842+0000", 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2576e-05 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.144690+0000", 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001847841 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.144708+0000", 2024-04-20T02:50:45.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7986999999999999e-05 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:133 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'86)", 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.479206+0000", 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.809529133000002, 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040552940000000001, 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479206+0000", 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479206+0000", 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479205+0000", 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479212+0000", 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1590000000000004e-06 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479222+0000", 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5920000000000007e-06 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479227+0000", 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2730000000000002e-06 2024-04-20T02:50:45.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479250+0000", 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2872999999999999e-05 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479261+0000", 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0818999999999999e-05 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483242+0000", 2024-04-20T02:50:45.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039811780000000001 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483261+0000", 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9018e-05 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:134 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'86)", 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.479634+0000", 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.809100612999998, 2024-04-20T02:50:45.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0036374179999999999, 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479634+0000", 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479634+0000", 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479633+0000", 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479671+0000", 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7889000000000003e-05 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479697+0000", 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5454999999999999e-05 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479702+0000", 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8010000000000003e-06 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479741+0000", 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9075000000000001e-05 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.479751+0000", 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0661e-05 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483262+0000", 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003510614 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483271+0000", 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5519999999999993e-06 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.328 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:135 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'86)", 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.480079+0000", 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.808655912999999, 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0032009180000000001, 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480079+0000", 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480079+0000", 2024-04-20T02:50:45.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480078+0000", 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480083+0000", 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2979999999999999e-06 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480096+0000", 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2926e-05 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480102+0000", 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.31e-06 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480124+0000", 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2220000000000001e-05 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480135+0000", 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1164e-05 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483272+0000", 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031368989999999999 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483280+0000", 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7190000000000001e-06 2024-04-20T02:50:45.332 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:136 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'86)", 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.480416+0000", 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.808318599, 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003224785, 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480416+0000", 2024-04-20T02:50:45.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480416+0000", 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480416+0000", 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480419+0000", 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1709999999999998e-06 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480424+0000", 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8679999999999998e-06 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480427+0000", 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6799999999999999e-06 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480450+0000", 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3042000000000001e-05 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.480463+0000", 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2377e-05 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483628+0000", 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031652289999999999 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483641+0000", 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2764e-05 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:141 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'87)", 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.483058+0000", 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.805677023000001, 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031466419999999998, 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483058+0000", 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483058+0000", 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483057+0000", 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483062+0000", 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1019999999999996e-06 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483070+0000", 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5669999999999997e-06 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483074+0000", 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7369999999999997e-06 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483108+0000", 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3399999999999999e-05 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.483119+0000", 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0756e-05 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.486176+0000", 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0030574320000000001 2024-04-20T02:50:45.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.486204+0000", 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8229e-05 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:144 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'102, mlcod=16'98)", 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.491197+0000", 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.797537363, 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0055184850000000001, 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.341 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491197+0000", 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491197+0000", 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491197+0000", 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491262+0000", 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4984e-05 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491269+0000", 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9310000000000002e-06 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491273+0000", 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8820000000000003e-06 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491296+0000", 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3238999999999999e-05 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491309+0000", 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2729e-05 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.496690+0000", 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053810489999999997 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.496716+0000", 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6069999999999999e-05 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:145 1.0 e16/15 1:30306672:devicehealth::main.db.0000000000000000:head v 16'103, mlcod=16'98)", 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.491959+0000", 2024-04-20T02:50:45.345 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.796776025, 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0047703800000000003, 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491959+0000", 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491959+0000", 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.491958+0000", 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.492035+0000", 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6979999999999998e-05 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.492041+0000", 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1340000000000001e-06 2024-04-20T02:50:45.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.492045+0000", 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0049999999999998e-06 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.492070+0000", 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4272000000000001e-05 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.492083+0000", 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3791999999999999e-05 2024-04-20T02:50:45.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.496717+0000", 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046339049999999998 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.496729+0000", 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1637999999999999e-05 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:146 1.0 e16/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'104, mlcod=16'102)", 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.497330+0000", 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.791404534000002, 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0035693740000000002, 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497330+0000", 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497330+0000", 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497329+0000", 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497343+0000", 2024-04-20T02:50:45.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3930999999999999e-05 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497353+0000", 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5659999999999996e-06 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497358+0000", 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0340000000000004e-06 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497391+0000", 2024-04-20T02:50:45.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2957999999999999e-05 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.497405+0000", 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3991e-05 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.500872+0000", 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034673960000000002 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.500899+0000", 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.711e-05 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:155 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'109, mlcod=16'105)", 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.952207+0000", 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.336527264000001, 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0035612809999999999, 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952207+0000", 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952207+0000", 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952207+0000", 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952315+0000", 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108133 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952322+0000", 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1080000000000004e-06 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952327+0000", 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3019999999999996e-06 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952350+0000", 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3011e-05 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952364+0000", 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4046000000000001e-05 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955738+0000", 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033740179999999999 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955769+0000", 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1078000000000003e-05 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:156 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'110, mlcod=16'105)", 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.952957+0000", 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.335777517, 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053524920000000004, 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.358 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952957+0000", 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952957+0000", 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952957+0000", 2024-04-20T02:50:45.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952960+0000", 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3730000000000001e-06 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952966+0000", 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9120000000000003e-06 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952970+0000", 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6990000000000001e-06 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.952996+0000", 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6084e-05 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953008+0000", 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2625e-05 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958283+0000", 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052744899999999997 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958310+0000", 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6693e-05 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:157 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'111, mlcod=16'105)", 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.953378+0000", 2024-04-20T02:50:45.362 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.335356389000001, 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0049499729999999999, 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953378+0000", 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953378+0000", 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953378+0000", 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953383+0000", 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8509999999999998e-06 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953388+0000", 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5339999999999999e-06 2024-04-20T02:50:45.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953391+0000", 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9869999999999999e-06 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953416+0000", 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4312e-05 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953429+0000", 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3550000000000001e-05 2024-04-20T02:50:45.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958312+0000", 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0048824610000000003 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958328+0000", 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6578999999999998e-05 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:158 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'112, mlcod=16'105)", 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.953823+0000", 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.334911375000001, 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0045220779999999997, 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953823+0000", 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953823+0000", 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953823+0000", 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953874+0000", 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0890999999999998e-05 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953879+0000", 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7370000000000003e-06 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953884+0000", 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0339999999999998e-06 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953907+0000", 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3059000000000001e-05 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.953921+0000", 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3985000000000001e-05 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958329+0000", 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0044084939999999998 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.958345+0000", 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6277999999999999e-05 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4201.0:159 1.0 e17/15 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=16'105)", 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:20.955539+0000", 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.333195777, 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056231900000000001, 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955539+0000", 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955539+0000", 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955538+0000", 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955544+0000", 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4779999999999999e-06 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955554+0000", 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.815e-06 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955561+0000", 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6000000000000001e-06 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955594+0000", 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2838e-05 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.955606+0000", 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1843e-05 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.961130+0000", 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0055239019999999998 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:20.961162+0000", 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2335000000000001e-05 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.374 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.375 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.375 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:45.375 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:45.375 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:45.375 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:50:45.377 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:50:45.377 INFO:tasks.rados.rados.0.smithi105.stdout:652: done (3 left) 2024-04-20T02:50:45.378 INFO:tasks.rados.rados.0.smithi105.stdout:653: done (2 left) 2024-04-20T02:50:45.378 INFO:tasks.rados.rados.0.smithi105.stdout:654: done (1 left) 2024-04-20T02:50:45.378 INFO:tasks.rados.rados.0.smithi105.stdout:655: done (0 left) 2024-04-20T02:50:45.378 INFO:tasks.rados.rados.0.smithi105.stdout:656: set_chunk oid 27 target oid 2024-04-20T02:50:45.378 INFO:tasks.rados.rados.0.smithi105.stdout:656: set_chunk oid 27 offset: 870400 length: 1270272 target oid 5520356cbad01cbb77690f4acae83315abf571d3bc9027883d585e07c7f7b8e1 offset: 0 snap -1 2024-04-20T02:50:45.378 INFO:tasks.rados.rados.0.smithi105.stdout:657: snap_remove snap 0 2024-04-20T02:50:45.378 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:45.377+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:45.478 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:45.478+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:45.494 INFO:teuthology.orchestra.run.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e25 pg_nums {1={14=1},2={18=8},3={21=4},4={22=16}} deleted_pools ) 2024-04-20T02:50:45.495 INFO:teuthology.orchestra.run.smithi138.stdout:Importing pgid 4.2 2024-04-20T02:50:45.495 INFO:teuthology.orchestra.run.smithi138.stdout:write_pg epoch 23 info 4.2( empty local-lis/les=22/23 n=0 ec=22/22 lis/c=22/22 les/c/f=23/23/0 sis=22) 2024-04-20T02:50:45.498 INFO:teuthology.orchestra.run.smithi138.stdout:Import successful 2024-04-20T02:50:45.551 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:45.551 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:45.551 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:50:45.551 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:45.562 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:50:45.603 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:45.603+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:45.679 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:45.679+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:45.732 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:45.732 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:45.732 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:50:45.732 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:50:45.732 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:45.742 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:50:45.786 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:45.786+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:45.870 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:45.870 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:50:45.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:50:45.870 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-04-20T02:50:45.870 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:45.880 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:50:45.921 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:45.921+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:45.982 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:45.983+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:46.011 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /var/log/ceph/exp.4.2.5 2024-04-20T02:50:46.048 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:46.048 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:46.048 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:50:46.048 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:46.058 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:50:46.084 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.5 2024-04-20T02:50:46.084 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-5 show-label 2024-04-20T02:50:46.099 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:46.099+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:46.173 INFO:teuthology.orchestra.run.smithi138.stdout:inferring bluefs devices from bluestore path 2024-04-20T02:50:46.174 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:46.174 INFO:teuthology.orchestra.run.smithi138.stdout: "/var/lib/ceph/osd/ceph-5/block": { 2024-04-20T02:50:46.174 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_uuid": "3e1c33fa-6c8c-4898-99bc-462ecef25f30", 2024-04-20T02:50:46.174 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 96636764160, 2024-04-20T02:50:46.174 INFO:teuthology.orchestra.run.smithi138.stdout: "btime": "2024-04-20T02:49:50.153702+0000", 2024-04-20T02:50:46.174 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "main", 2024-04-20T02:50:46.174 INFO:teuthology.orchestra.run.smithi138.stdout: "bfm_blocks": "23592960", 2024-04-20T02:50:46.174 INFO:teuthology.orchestra.run.smithi138.stdout: "bfm_blocks_per_key": "128", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "bfm_bytes_per_block": "4096", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "bfm_size": "96636764160", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs": "1", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_fsid": "a524a365-eaec-4ebd-aa4d-0d76d55a252f", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_version_when_created": "ceph version 18.2.2-1252-g15eecde2 (15eecde2bab9803889d929c7e423d07970d8cbed) reef (stable)", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "created_at": "2024-04-20T02:49:52.484825Z", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "kv_backend": "rocksdb", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "magic": "ceph osd volume v026", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "mkfs_done": "yes", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "ready": "ready", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "require_osd_release": "18", 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: "whoami": "5" 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:46.175 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:46.176 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.5 2024-04-20T02:50:46.177 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-5 show-sharding 2024-04-20T02:50:46.222 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:46.221+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:46.225 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:46.225 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:46.225 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:50:46.225 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:50:46.225 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:46.236 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:50:46.285 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:46.285+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:46.323 INFO:tasks.rados.rados.0.smithi105.stdout:657: done (1 left) 2024-04-20T02:50:46.323 INFO:tasks.rados.rados.0.smithi105.stdout:658: tier_evict oid 8 2024-04-20T02:50:46.323 INFO:tasks.rados.rados.0.smithi105.stdout:658: tier_evict oid 8 snap -1 2024-04-20T02:50:46.323 INFO:tasks.rados.rados.0.smithi105.stdout:659: rollback oid 35 current snap is 1 2024-04-20T02:50:46.323 INFO:tasks.rados.rados.0.smithi105.stdout:659: done (2 left) 2024-04-20T02:50:46.324 INFO:tasks.rados.rados.0.smithi105.stdout:660: set_chunk oid 22 target oid 2024-04-20T02:50:46.327 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:50:46.336 INFO:tasks.rados.rados.0.smithi105.stdout:660: set_chunk oid 22 offset: 1438208 length: 500736 target oid c94bb7525451dafb1c7574337e72ec8a847284b2cbe33f8de068e3e5d4351319 offset: 0 snap -1 2024-04-20T02:50:46.336 INFO:tasks.rados.rados.0.smithi105.stdout:661: write oid 10 current snap is 1 2024-04-20T02:50:46.336 INFO:tasks.rados.rados.0.smithi105.stdout:656: finishing set_chunk to oid 27 2024-04-20T02:50:46.336 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 870400 length: 1270272 2024-04-20T02:50:46.336 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-20T02:50:46.336 INFO:tasks.rados.rados.0.smithi105.stdout:658: got (22) Invalid argument 2024-04-20T02:50:46.336 INFO:tasks.rados.rados.0.smithi105.stdout:661: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-04-20T02:50:46.341 INFO:tasks.rados.rados.0.smithi105.stdout:661: writing smithi10548823-10 from 410104 to 875209 tid 1 2024-04-20T02:50:46.349 INFO:tasks.rados.rados.0.smithi105.stdout:661: writing smithi10548823-10 from 1563527 to 2146552 tid 2 2024-04-20T02:50:46.356 INFO:tasks.rados.rados.0.smithi105.stdout:661: writing smithi10548823-10 from 2810926 to 3210104 tid 3 2024-04-20T02:50:46.356 INFO:tasks.rados.rados.0.smithi105.stdout:656: done (3 left) 2024-04-20T02:50:46.356 INFO:tasks.rados.rados.0.smithi105.stdout:658: done (2 left) 2024-04-20T02:50:46.356 INFO:tasks.rados.rados.0.smithi105.stdout:662: write oid 4 current snap is 1 2024-04-20T02:50:46.356 INFO:tasks.rados.rados.0.smithi105.stdout:660: finishing set_chunk to oid 22 2024-04-20T02:50:46.356 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 22 set_chunk returned error code -95 offset: 1438208 length: 500736 2024-04-20T02:50:46.356 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-20T02:50:46.356 INFO:tasks.rados.rados.0.smithi105.stdout:662: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-04-20T02:50:46.361 INFO:tasks.rados.rados.0.smithi105.stdout:662: writing smithi10548823-4 from 426911 to 967265 tid 1 2024-04-20T02:50:46.370 INFO:tasks.rados.rados.0.smithi105.stdout:662: writing smithi10548823-4 from 1622009 to 2348692 tid 2 2024-04-20T02:50:46.376 INFO:tasks.rados.rados.0.smithi105.stdout:662: writing smithi10548823-4 from 3121996 to 3226911 tid 3 2024-04-20T02:50:46.376 INFO:tasks.rados.rados.0.smithi105.stdout:660: done (2 left) 2024-04-20T02:50:46.376 INFO:tasks.rados.rados.0.smithi105.stdout:663: set_chunk oid 47 target oid 2024-04-20T02:50:46.385 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:46.385+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:46.387 INFO:tasks.rados.rados.0.smithi105.stdout:663: set_chunk oid 47 offset: 2152960 length: 219136 target oid 3d7388f1d721e6c9b078be5bdbcdcb756b953def85720bebbae38eeecb7fd08e offset: 0 snap -1 2024-04-20T02:50:46.387 INFO:tasks.rados.rados.0.smithi105.stdout:664: read oid 49 snap -1 2024-04-20T02:50:46.387 INFO:tasks.rados.rados.0.smithi105.stdout:664: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-20T02:50:46.387 INFO:tasks.rados.rados.0.smithi105.stdout:665: write oid 34 current snap is 1 2024-04-20T02:50:46.387 INFO:tasks.rados.rados.0.smithi105.stdout:661: finishing write tid 1 to smithi10548823-10 2024-04-20T02:50:46.387 INFO:tasks.rados.rados.0.smithi105.stdout:661: finishing write tid 2 to smithi10548823-10 2024-04-20T02:50:46.387 INFO:tasks.rados.rados.0.smithi105.stdout:661: finishing write tid 3 to smithi10548823-10 2024-04-20T02:50:46.388 INFO:tasks.rados.rados.0.smithi105.stdout:661: finishing write tid 4 to smithi10548823-10 2024-04-20T02:50:46.388 INFO:tasks.rados.rados.0.smithi105.stdout:665: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-04-20T02:50:46.390 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:46.390 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:50:46.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:50:46.390 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-04-20T02:50:46.390 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:46.393 INFO:tasks.rados.rados.0.smithi105.stdout:665: writing smithi10548823-34 from 443718 to 1059321 tid 1 2024-04-20T02:50:46.399 INFO:tasks.rados.rados.0.smithi105.stdout:665: writing smithi10548823-34 from 1564138 to 2034479 tid 2 2024-04-20T02:50:46.405 INFO:tasks.rados.rados.0.smithi105.stdout:665: writing smithi10548823-34 from 2633066 to 3243718 tid 3 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:666: write (excl) oid 44 current snap is 1 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:661: finishing write tid 5 to smithi10548823-10 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:661: oid 10 updating version 0 to 36 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:661: oid 10 updating version 36 to 38 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:661: oid 10 updating version 38 to 39 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:661: oid 10 version 39 is already newer than 37 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 39 (ObjNum 72 snap 1 seq_num 72) dirty exists 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:661: left oid 10 (ObjNum 72 snap 1 seq_num 72) 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:663: finishing set_chunk to oid 47 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 2152960 length: 219136 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:46.406 INFO:tasks.rados.rados.0.smithi105.stdout:664: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-20T02:50:46.407 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:50:46.492 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:46.492+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:46.510 INFO:tasks.rados.rados.0.smithi105.stdout:662: finishing write tid 1 to smithi10548823-4 2024-04-20T02:50:46.510 INFO:tasks.rados.rados.0.smithi105.stdout:662: finishing write tid 2 to smithi10548823-4 2024-04-20T02:50:46.510 INFO:tasks.rados.rados.0.smithi105.stdout:666: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-04-20T02:50:46.515 INFO:tasks.rados.rados.0.smithi105.stdout:666: writing smithi10548823-44 from 460525 to 1151377 tid 1 2024-04-20T02:50:46.522 INFO:tasks.rados.rados.0.smithi105.stdout:666: writing smithi10548823-44 from 1622620 to 2352972 tid 2 2024-04-20T02:50:46.527 INFO:tasks.rados.rados.0.smithi105.stdout:666: writing smithi10548823-44 from 3060489 to 3260525 tid 3 2024-04-20T02:50:46.527 INFO:tasks.rados.rados.0.smithi105.stdout:661: done (5 left) 2024-04-20T02:50:46.527 INFO:tasks.rados.rados.0.smithi105.stdout:663: done (4 left) 2024-04-20T02:50:46.528 INFO:tasks.rados.rados.0.smithi105.stdout:664: done (3 left) 2024-04-20T02:50:46.528 INFO:tasks.rados.rados.0.smithi105.stdout:667: read oid 38 snap -1 2024-04-20T02:50:46.528 INFO:tasks.rados.rados.0.smithi105.stdout:667: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:50:46.528 INFO:tasks.rados.rados.0.smithi105.stdout:668: set_chunk oid 42 target oid 2024-04-20T02:50:46.533 INFO:tasks.rados.rados.0.smithi105.stdout:668: set_chunk oid 42 offset: 1756672 length: 75264 target oid 126d50b9ccd058128accdb5f9e4984ecfbb65aa7997f1616d3dcd0c09fefc945 offset: 0 snap -1 2024-04-20T02:50:46.533 INFO:tasks.rados.rados.0.smithi105.stdout:669: read oid 26 snap -1 2024-04-20T02:50:46.533 INFO:tasks.rados.rados.0.smithi105.stdout:669: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-20T02:50:46.533 INFO:tasks.rados.rados.0.smithi105.stdout:670: read oid 12 snap -1 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:670: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:671: read oid 50 snap -1 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:671: expect (ObjNum 49 snap 0 seq_num 49) 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:662: finishing write tid 3 to smithi10548823-4 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:672: read oid 24 snap -1 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:672: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:673: tier_evict oid 49 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:673: tier_evict oid 49 snap -1 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:674: read oid 45 snap -1 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:674: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-20T02:50:46.534 INFO:tasks.rados.rados.0.smithi105.stdout:675: write (excl) oid 39 current snap is 1 2024-04-20T02:50:46.535 INFO:tasks.rados.rados.0.smithi105.stdout:675: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-04-20T02:50:46.542 INFO:tasks.rados.rados.0.smithi105.stdout:675: writing smithi10548823-39 from 477332 to 1243433 tid 1 2024-04-20T02:50:46.548 INFO:tasks.rados.rados.0.smithi105.stdout:675: writing smithi10548823-39 from 1681102 to 2155112 tid 2 2024-04-20T02:50:46.557 INFO:tasks.rados.rados.0.smithi105.stdout:675: writing smithi10548823-39 from 2687912 to 3277332 tid 3 2024-04-20T02:50:46.557 INFO:tasks.rados.rados.0.smithi105.stdout:676: write (excl) oid 10 current snap is 1 2024-04-20T02:50:46.557 INFO:tasks.rados.rados.0.smithi105.stdout:676: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-04-20T02:50:46.563 INFO:tasks.rados.rados.0.smithi105.stdout:676: writing smithi10548823-10 from 494139 to 1051842 tid 1 2024-04-20T02:50:46.570 INFO:tasks.rados.rados.0.smithi105.stdout:676: writing smithi10548823-10 from 1455937 to 2189958 tid 2 2024-04-20T02:50:46.577 INFO:tasks.rados.rados.0.smithi105.stdout:676: writing smithi10548823-10 from 2948041 to 3294139 tid 3 2024-04-20T02:50:46.577 INFO:tasks.rados.rados.0.smithi105.stdout:677: rollback oid 37 current snap is 1 2024-04-20T02:50:46.577 INFO:tasks.rados.rados.0.smithi105.stdout:677: done (13 left) 2024-04-20T02:50:46.577 INFO:tasks.rados.rados.0.smithi105.stdout:678: read oid 2 snap -1 2024-04-20T02:50:46.577 INFO:tasks.rados.rados.0.smithi105.stdout:678: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-20T02:50:46.577 INFO:tasks.rados.rados.0.smithi105.stdout:679: write oid 30 current snap is 1 2024-04-20T02:50:46.577 INFO:tasks.rados.rados.0.smithi105.stdout:679: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-04-20T02:50:46.583 INFO:tasks.rados.rados.0.smithi105.stdout:679: writing smithi10548823-30 from 510946 to 1143898 tid 1 2024-04-20T02:50:46.587 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:46.587+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:46.590 INFO:tasks.rados.rados.0.smithi105.stdout:679: writing smithi10548823-30 from 1798066 to 2275745 tid 2 2024-04-20T02:50:46.596 INFO:tasks.rados.rados.0.smithi105.stdout:679: writing smithi10548823-30 from 2742758 to 3199975 tid 3 2024-04-20T02:50:46.596 INFO:tasks.rados.rados.0.smithi105.stdout:680: write (excl) oid 29 current snap is 1 2024-04-20T02:50:46.596 INFO:tasks.rados.rados.0.smithi105.stdout:680: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-04-20T02:50:46.604 INFO:tasks.rados.rados.0.smithi105.stdout:680: writing smithi10548823-29 from 527753 to 1235954 tid 1 2024-04-20T02:50:46.612 INFO:tasks.rados.rados.0.smithi105.stdout:680: writing smithi10548823-29 from 1856548 to 2594238 tid 2 2024-04-20T02:50:46.614 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:46.614 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:46.614 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:50:46.614 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:680: writing smithi10548823-29 from 3286534 to 3327753 tid 3 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:662: finishing write tid 4 to smithi10548823-4 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:662: finishing write tid 5 to smithi10548823-4 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:662: oid 4 updating version 0 to 48 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:662: oid 4 version 48 is already newer than 47 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:662: oid 4 version 48 is already newer than 46 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:662: oid 4 version 48 is already newer than 45 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 48 (ObjNum 73 snap 1 seq_num 73) dirty exists 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:662: left oid 4 (ObjNum 73 snap 1 seq_num 73) 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:665: finishing write tid 1 to smithi10548823-34 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:665: finishing write tid 2 to smithi10548823-34 2024-04-20T02:50:46.617 INFO:tasks.rados.rados.0.smithi105.stdout:665: finishing write tid 3 to smithi10548823-34 2024-04-20T02:50:46.618 INFO:tasks.rados.rados.0.smithi105.stdout:665: finishing write tid 4 to smithi10548823-34 2024-04-20T02:50:46.618 INFO:tasks.rados.rados.0.smithi105.stdout:665: finishing write tid 5 to smithi10548823-34 2024-04-20T02:50:46.618 INFO:tasks.rados.rados.0.smithi105.stdout:665: oid 34 updating version 0 to 50 2024-04-20T02:50:46.618 INFO:tasks.rados.rados.0.smithi105.stdout:665: oid 34 updating version 50 to 51 2024-04-20T02:50:46.618 INFO:tasks.rados.rados.0.smithi105.stdout:665: oid 34 version 51 is already newer than 49 2024-04-20T02:50:46.618 INFO:tasks.rados.rados.0.smithi105.stdout:665: oid 34 updating version 51 to 52 2024-04-20T02:50:46.618 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 52 (ObjNum 74 snap 1 seq_num 74) dirty exists 2024-04-20T02:50:46.618 INFO:tasks.rados.rados.0.smithi105.stdout:665: left oid 34 (ObjNum 74 snap 1 seq_num 74) 2024-04-20T02:50:46.618 INFO:tasks.rados.rados.0.smithi105.stdout:662: done (15 left) 2024-04-20T02:50:46.619 INFO:tasks.rados.rados.0.smithi105.stdout:665: done (14 left) 2024-04-20T02:50:46.619 INFO:tasks.rados.rados.0.smithi105.stdout:681: write (excl) oid 6 current snap is 1 2024-04-20T02:50:46.619 INFO:tasks.rados.rados.0.smithi105.stdout:673: got (22) Invalid argument 2024-04-20T02:50:46.619 INFO:tasks.rados.rados.0.smithi105.stdout:667: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:50:46.626 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:50:46.714 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:46.712+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:46.715 INFO:tasks.rados.rados.0.smithi105.stdout:668: finishing set_chunk to oid 42 2024-04-20T02:50:46.715 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk returned error code -95 offset: 1756672 length: 75264 2024-04-20T02:50:46.715 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:46.715 INFO:tasks.rados.rados.0.smithi105.stdout:681: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-04-20T02:50:46.722 INFO:tasks.rados.rados.0.smithi105.stdout:681: writing smithi10548823-6 from 544560 to 1328010 tid 1 2024-04-20T02:50:46.729 INFO:tasks.rados.rados.0.smithi105.stdout:681: writing smithi10548823-6 from 1915030 to 2396378 tid 2 2024-04-20T02:50:46.735 INFO:tasks.rados.rados.0.smithi105.stdout:681: writing smithi10548823-6 from 2797604 to 3277365 tid 3 2024-04-20T02:50:46.735 INFO:tasks.rados.rados.0.smithi105.stdout:667: done (14 left) 2024-04-20T02:50:46.735 INFO:tasks.rados.rados.0.smithi105.stdout:668: done (13 left) 2024-04-20T02:50:46.735 INFO:tasks.rados.rados.0.smithi105.stdout:673: done (12 left) 2024-04-20T02:50:46.735 INFO:tasks.rados.rados.0.smithi105.stdout:682: read oid 34 snap -1 2024-04-20T02:50:46.735 INFO:tasks.rados.rados.0.smithi105.stdout:682: expect (ObjNum 74 snap 1 seq_num 74) 2024-04-20T02:50:46.736 INFO:tasks.rados.rados.0.smithi105.stdout:683: set_chunk oid 42 target oid 2024-04-20T02:50:46.741 INFO:tasks.rados.rados.0.smithi105.stdout:683: set_chunk oid 42 offset: 10752 length: 552960 target oid d1768af9860251b5b7bc5c43b5368761bba981c92dd8005238016b83940d5bad offset: 0 snap -1 2024-04-20T02:50:46.741 INFO:tasks.rados.rados.0.smithi105.stdout:684: tier_flush oid 21 2024-04-20T02:50:46.741 INFO:tasks.rados.rados.0.smithi105.stdout:684: tier_flush oid 21 snap -1 2024-04-20T02:50:46.741 INFO:tasks.rados.rados.0.smithi105.stdout:670: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-20T02:50:46.789 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:46.789+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:46.820 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:46.820 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:46.821 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:50:46.821 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:50:46.821 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:46.831 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:50:46.889 INFO:tasks.rados.rados.0.smithi105.stdout:674: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-20T02:50:46.889 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:46.889+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:46.980 INFO:tasks.rados.rados.0.smithi105.stdout:671: expect (ObjNum 49 snap 0 seq_num 49) 2024-04-20T02:50:46.985 INFO:teuthology.orchestra.run.smithi138.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-04-20T02:50:46.989 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.5 2024-04-20T02:50:46.989 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-04-20T02:50:46.989 INFO:tasks.thrashosds.thrasher:applying new sharding: O(4) P(4,8-) L(5) 2024-04-20T02:50:46.990 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-5 --sharding 'O(4) P(4,8-) L(5)' reshard 2024-04-20T02:50:46.991 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:46.991+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:47.002 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:913 3.2 3:6ff86bb8:::5520356cbad01cbb77690f4acae83315abf571d3bc9027883d585e07c7f7b8e1:head [write 0~1270272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:45.323476+0000", 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 1.6788596259999999, 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018063643000000001, 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:47.003 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 913 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.323476+0000", 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.323476+0000", 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.323474+0000", 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.323982+0000", 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00050864499999999995 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.323997+0000", 2024-04-20T02:50:47.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4959e-05 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.324005+0000", 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1419999999999993e-06 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.324060+0000", 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4367000000000003e-05 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.324171+0000", 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111019 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.324275+0000", 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104088 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.337406+0000", 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013131230000000001 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.339254+0000", 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018480669999999999 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.341469+0000", 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00221456 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.341498+0000", 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9437e-05 2024-04-20T02:50:47.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.341539+0000", 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1211999999999998e-05 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:941 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'19, mlcod=30'17)", 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.484044215, 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042482077, 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.009 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518290+0000", 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518988+0000", 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069878099999999999 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519003+0000", 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4756999999999999e-05 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519013+0000", 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4250000000000004e-06 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519071+0000", 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8628000000000002e-05 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519106+0000", 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4817000000000002e-05 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560724+0000", 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041618334 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560773+0000", 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8937999999999998e-05 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.013 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:942 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'20, mlcod=30'17)", 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.465227106, 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030979784999999999, 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:47.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537107+0000", 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538354+0000", 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001247232 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538381+0000", 2024-04-20T02:50:47.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7008999999999998e-05 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538395+0000", 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4093999999999999e-05 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538476+0000", 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1045999999999996e-05 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538505+0000", 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8969999999999999e-05 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568053+0000", 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029547370999999999 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568088+0000", 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5540000000000002e-05 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.017 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.45859588000000001, 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:50:47.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:50:47.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:50:47.023 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.45237480600000002, 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:50:47.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:50:47.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:50:47.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:50:47.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:50:47.030 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.030 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.030 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.030 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:943 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'21, mlcod=30'17)", 2024-04-20T02:50:47.030 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.553675+0000", 2024-04-20T02:50:47.030 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.44866076900000001, 2024-04-20T02:50:47.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014930229, 2024-04-20T02:50:47.030 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553675+0000", 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553675+0000", 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553673+0000", 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553951+0000", 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027789900000000002 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553962+0000", 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0855e-05 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553970+0000", 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1470000000000001e-06 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560816+0000", 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068456460000000004 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560851+0000", 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4555999999999999e-05 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568569+0000", 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.007718478 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568605+0000", 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5839e-05 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.443974602, 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:50:47.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:50:47.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:50:47.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:50:47.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:50:47.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:50:47.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:971 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (56) in=7b,truncate 3277332] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.44365653300000002, 2024-04-20T02:50:47.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051333776999999997, 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 971 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558678+0000", 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558686+0000", 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7330000000000003e-06 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558693+0000", 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.866e-06 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558698+0000", 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.49e-06 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594685+0000", 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035986753000000003 2024-04-20T02:50:47.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594792+0000", 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107129 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594958+0000", 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000165429 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595721+0000", 2024-04-20T02:50:47.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076360100000000004 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.596242+0000", 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052083800000000003 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609915+0000", 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013673431 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609958+0000", 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2759000000000002e-05 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610013+0000", 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4421e-05 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:47.049 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.44363195700000002, 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:50:47.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:50:47.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:944 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'22, mlcod=30'17)", 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.559099+0000", 2024-04-20T02:50:47.054 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.44323681100000001, 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0098123140000000008, 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559099+0000", 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559099+0000", 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559098+0000", 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559106+0000", 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.028e-06 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559116+0000", 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9310000000000003e-06 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559122+0000", 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1299999999999998e-06 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.567846+0000", 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0087244330000000002 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.567871+0000", 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4865e-05 2024-04-20T02:50:47.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568877+0000", 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010063870000000001 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568911+0000", 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3458999999999999e-05 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.058 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop_reply(client.4431.0:968 4.0 e32/22 ondisk, result = 0)", 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.43619167199999997, 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028443566999999999, 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566143+0000", 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566152+0000", 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1759999999999999e-06 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566163+0000", 2024-04-20T02:50:47.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0852999999999999e-05 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566172+0000", 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2229999999999996e-06 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594564+0000", 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028392473000000001 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594574+0000", 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5899999999999997e-06 2024-04-20T02:50:47.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594587+0000", 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3454e-05 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop_reply(client.4431.0:969 4.0 e32/22 ondisk, result = 0)", 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.578344+0000", 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.42399150899999999, 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016282119000000001, 2024-04-20T02:50:47.062 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578344+0000", 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578344+0000", 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578343+0000", 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578354+0000", 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1241e-05 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578370+0000", 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6668999999999999e-05 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578383+0000", 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2330999999999999e-05 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594611+0000", 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01622852 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594618+0000", 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0679999999999999e-06 2024-04-20T02:50:47.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594626+0000", 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6729999999999992e-06 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:986 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'5, mlcod=29'3)", 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.366130915, 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.040291053, 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.066 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636467+0000", 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026249800000000001 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636478+0000", 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1762e-05 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636493+0000", 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4826e-05 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636546+0000", 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2638e-05 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636570+0000", 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4235000000000001e-05 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676457+0000", 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039887226999999997 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676496+0000", 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8281000000000002e-05 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:987 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'6, mlcod=29'3)", 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.651660+0000", 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.35067516900000001, 2024-04-20T02:50:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026492596, 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651660+0000", 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651660+0000", 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651659+0000", 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651882+0000", 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000222222 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651896+0000", 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4541e-05 2024-04-20T02:50:47.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651905+0000", 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1260000000000004e-06 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.657994+0000", 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0060883889999999996 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.658012+0000", 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7912000000000001e-05 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678124+0000", 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020112623999999999 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678153+0000", 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8632e-05 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:988 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'7, mlcod=29'3)", 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.667378+0000", 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.33495719499999999, 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011143905000000001, 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667378+0000", 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667378+0000", 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667377+0000", 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667399+0000", 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.209e-05 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667410+0000", 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0906e-05 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667418+0000", 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7430000000000002e-06 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676540+0000", 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0091222109999999999 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676565+0000", 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5196999999999999e-05 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678498+0000", 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00193249 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678522+0000", 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4414999999999999e-05 2024-04-20T02:50:47.078 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:989 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'8, mlcod=29'3)", 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.668591+0000", 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.333744031, 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010451993, 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668591+0000", 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668591+0000", 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668591+0000", 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668597+0000", 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3589999999999996e-06 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668611+0000", 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3519e-05 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668617+0000", 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7839999999999998e-06 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677949+0000", 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093313839999999999 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677973+0000", 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4127000000000001e-05 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679020+0000", 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001046644 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679043+0000", 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3771e-05 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1002 4.0 4:0158e114:::smithi10548823-21:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.25989710100000002, 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042474603999999999, 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.083 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1002 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:47.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742436+0000", 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742449+0000", 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3042e-05 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742469+0000", 2024-04-20T02:50:47.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0007999999999999e-05 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742481+0000", 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1297999999999999e-05 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742528+0000", 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7697000000000001e-05 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742598+0000", 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9638e-05 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.784913+0000", 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042314928000000002 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.3.13:30 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.750440+0000", 2024-04-20T02:50:47.087 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.25189497700000002, 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018760754000000001, 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750440+0000", 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750440+0000", 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750439+0000", 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750701+0000", 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026198200000000001 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750716+0000", 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5128e-05 2024-04-20T02:50:47.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750727+0000", 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1386000000000001e-05 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750782+0000", 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4985000000000003e-05 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750944+0000", 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016174000000000001 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751215+0000", 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027062199999999998 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756304+0000", 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050896700000000001 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:47.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.757875+0000", 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001570788 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.769115+0000", 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011240333999999999 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.769165+0000", 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9784000000000003e-05 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.769201+0000", 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6004999999999998e-05 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.3.13:32 3.1 e32/21 3:aa868ed7:::8b42333603b5d66a962ab75ee4beb7799a2124b7175acbb12b87093577a68b6f:head v 32'86, mlcod=32'84)", 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.751861+0000", 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.25047465499999999, 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012698506, 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751861+0000", 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751861+0000", 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751860+0000", 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:47.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751933+0000", 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3002000000000003e-05 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751942+0000", 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9670000000000007e-06 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751949+0000", 2024-04-20T02:50:47.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4769999999999997e-06 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.752252+0000", 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030264000000000002 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.752272+0000", 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0468000000000001e-05 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.764519+0000", 2024-04-20T02:50:47.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012246309 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.764559+0000", 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0543000000000001e-05 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.3.13:33 3.1 e32/21 3:886a183d:::3dba94e9eda505a2c38254a500ade87922ef0a7ff12c5928203f64ac9e8a3c99:head v 32'87, mlcod=32'84)", 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.756413+0000", 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.24592238699999999, 2024-04-20T02:50:47.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0081679879999999993, 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756413+0000", 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756413+0000", 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756412+0000", 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756437+0000", 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4346999999999999e-05 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756465+0000", 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8867999999999999e-05 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756473+0000", 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.892e-06 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.760217+0000", 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037438910000000001 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.760240+0000", 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2980999999999999e-05 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.764562+0000", 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043219390000000003 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.764581+0000", 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8859e-05 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:47.101 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:47.102 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:50:47.104 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:50:47.104 INFO:tasks.rados.rados.0.smithi105.stdout:669: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-20T02:50:47.105 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:47.103+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:47.148 INFO:tasks.rados.rados.0.smithi105.stdout:672: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-20T02:50:47.203 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:47.203+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:47.232 INFO:tasks.rados.rados.0.smithi105.stdout:666: finishing write tid 1 to smithi10548823-44 2024-04-20T02:50:47.233 INFO:tasks.rados.rados.0.smithi105.stdout:666: finishing write tid 2 to smithi10548823-44 2024-04-20T02:50:47.233 INFO:tasks.rados.rados.0.smithi105.stdout:669: done (14 left) 2024-04-20T02:50:47.233 INFO:tasks.rados.rados.0.smithi105.stdout:670: done (13 left) 2024-04-20T02:50:47.233 INFO:tasks.rados.rados.0.smithi105.stdout:671: done (12 left) 2024-04-20T02:50:47.233 INFO:tasks.rados.rados.0.smithi105.stdout:672: done (11 left) 2024-04-20T02:50:47.233 INFO:tasks.rados.rados.0.smithi105.stdout:674: done (10 left) 2024-04-20T02:50:47.233 INFO:tasks.rados.rados.0.smithi105.stdout:685: write oid 1 current snap is 1 2024-04-20T02:50:47.233 INFO:tasks.rados.rados.0.smithi105.stdout:666: finishing write tid 3 to smithi10548823-44 2024-04-20T02:50:47.233 INFO:tasks.rados.rados.0.smithi105.stdout:685: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-04-20T02:50:47.236 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:47.236 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:47.236 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:47.236 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:47.238 INFO:tasks.rados.rados.0.smithi105.stdout:685: writing smithi10548823-1 from 561367 to 1020066 tid 1 2024-04-20T02:50:47.245 INFO:tasks.rados.rados.0.smithi105.stdout:685: writing smithi10548823-1 from 1573512 to 2314871 tid 2 2024-04-20T02:50:47.247 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:685: writing smithi10548823-1 from 2941380 to 3361367 tid 3 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:686: write oid 48 current snap is 1 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:666: finishing write tid 4 to smithi10548823-44 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:666: finishing write tid 5 to smithi10548823-44 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:666: oid 44 updating version 0 to 21 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:666: oid 44 updating version 21 to 22 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:666: oid 44 version 22 is already newer than 19 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:666: oid 44 version 22 is already newer than 20 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 22 (ObjNum 75 snap 1 seq_num 75) dirty exists 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:666: left oid 44 (ObjNum 75 snap 1 seq_num 75) 2024-04-20T02:50:47.252 INFO:tasks.rados.rados.0.smithi105.stdout:678: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-20T02:50:47.308 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:50:47.329 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:50:47.332 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:47.332+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:675: finishing write tid 1 to smithi10548823-39 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:675: finishing write tid 2 to smithi10548823-39 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:675: finishing write tid 3 to smithi10548823-39 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:675: finishing write tid 4 to smithi10548823-39 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:675: finishing write tid 5 to smithi10548823-39 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:675: oid 39 updating version 0 to 32 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:675: oid 39 updating version 32 to 34 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:675: oid 39 updating version 34 to 35 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:675: oid 39 version 35 is already newer than 33 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 35 (ObjNum 76 snap 1 seq_num 76) dirty exists 2024-04-20T02:50:47.338 INFO:tasks.rados.rados.0.smithi105.stdout:675: left oid 39 (ObjNum 76 snap 1 seq_num 76) 2024-04-20T02:50:47.339 INFO:tasks.rados.rados.0.smithi105.stdout:676: finishing write tid 1 to smithi10548823-10 2024-04-20T02:50:47.339 INFO:tasks.rados.rados.0.smithi105.stdout:676: finishing write tid 2 to smithi10548823-10 2024-04-20T02:50:47.339 INFO:tasks.rados.rados.0.smithi105.stdout:676: finishing write tid 3 to smithi10548823-10 2024-04-20T02:50:47.339 INFO:tasks.rados.rados.0.smithi105.stdout:686: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-04-20T02:50:47.344 INFO:tasks.rados.rados.0.smithi105.stdout:686: writing smithi10548823-48 from 578174 to 1112122 tid 1 2024-04-20T02:50:47.348 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:50:47.350 INFO:tasks.rados.rados.0.smithi105.stdout:686: writing smithi10548823-48 from 1515641 to 2000658 tid 2 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:686: writing smithi10548823-48 from 2736097 to 3238402 tid 3 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:666: done (11 left) 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:675: done (10 left) 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:678: done (9 left) 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:687: read oid 26 snap -1 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:687: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:688: write (excl) oid 18 current snap is 1 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:676: finishing write tid 4 to smithi10548823-10 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:676: finishing write tid 5 to smithi10548823-10 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:676: oid 10 updating version 0 to 40 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:676: oid 10 updating version 40 to 43 2024-04-20T02:50:47.359 INFO:tasks.rados.rados.0.smithi105.stdout:676: oid 10 version 43 is already newer than 42 2024-04-20T02:50:47.360 INFO:tasks.rados.rados.0.smithi105.stdout:676: oid 10 version 43 is already newer than 41 2024-04-20T02:50:47.360 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 43 (ObjNum 77 snap 1 seq_num 77) dirty exists 2024-04-20T02:50:47.360 INFO:tasks.rados.rados.0.smithi105.stdout:676: left oid 10 (ObjNum 77 snap 1 seq_num 77) 2024-04-20T02:50:47.360 INFO:tasks.rados.rados.0.smithi105.stdout:679: finishing write tid 1 to smithi10548823-30 2024-04-20T02:50:47.360 INFO:tasks.rados.rados.0.smithi105.stdout:679: finishing write tid 2 to smithi10548823-30 2024-04-20T02:50:47.360 INFO:tasks.rados.rados.0.smithi105.stdout:679: finishing write tid 3 to smithi10548823-30 2024-04-20T02:50:47.360 INFO:tasks.rados.rados.0.smithi105.stdout:679: finishing write tid 4 to smithi10548823-30 2024-04-20T02:50:47.360 INFO:tasks.rados.rados.0.smithi105.stdout:679: finishing write tid 5 to smithi10548823-30 2024-04-20T02:50:47.360 INFO:tasks.rados.rados.0.smithi105.stdout:679: oid 30 updating version 0 to 20 2024-04-20T02:50:47.361 INFO:tasks.rados.rados.0.smithi105.stdout:679: oid 30 updating version 20 to 22 2024-04-20T02:50:47.361 INFO:tasks.rados.rados.0.smithi105.stdout:679: oid 30 version 22 is already newer than 19 2024-04-20T02:50:47.361 INFO:tasks.rados.rados.0.smithi105.stdout:679: oid 30 version 22 is already newer than 21 2024-04-20T02:50:47.361 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 22 (ObjNum 78 snap 1 seq_num 78) dirty exists 2024-04-20T02:50:47.361 INFO:tasks.rados.rados.0.smithi105.stdout:679: left oid 30 (ObjNum 78 snap 1 seq_num 78) 2024-04-20T02:50:47.361 INFO:tasks.rados.rados.0.smithi105.stdout:680: finishing write tid 1 to smithi10548823-29 2024-04-20T02:50:47.361 INFO:tasks.rados.rados.0.smithi105.stdout:680: finishing write tid 2 to smithi10548823-29 2024-04-20T02:50:47.361 INFO:tasks.rados.rados.0.smithi105.stdout:680: finishing write tid 3 to smithi10548823-29 2024-04-20T02:50:47.361 INFO:tasks.rados.rados.0.smithi105.stdout:680: finishing write tid 4 to smithi10548823-29 2024-04-20T02:50:47.361 INFO:tasks.rados.rados.0.smithi105.stdout:680: finishing write tid 5 to smithi10548823-29 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:680: oid 29 updating version 0 to 8 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:680: oid 29 version 8 is already newer than 7 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:680: oid 29 version 8 is already newer than 6 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:680: oid 29 version 8 is already newer than 5 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 8 (ObjNum 79 snap 1 seq_num 79) dirty exists 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:680: left oid 29 (ObjNum 79 snap 1 seq_num 79) 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:683: finishing set_chunk to oid 42 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk returned error code -95 offset: 10752 length: 552960 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:681: finishing write tid 1 to smithi10548823-6 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:681: finishing write tid 2 to smithi10548823-6 2024-04-20T02:50:47.362 INFO:tasks.rados.rados.0.smithi105.stdout:682: expect (ObjNum 74 snap 1 seq_num 74) 2024-04-20T02:50:47.378 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:50:47.400 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:50:47.405 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:47.405+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:47.419 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:47.419 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:47.419 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:50:47.431 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:50:47.439 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:50:47.439 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:50:47.439 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:50:47.439 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:50:47.439 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:50:47.440 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:50:47.440 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-20T02:50:47.440 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:50:47.440 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:50:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:50:47.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:50:47.506 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:47.506+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:47.510 INFO:tasks.rados.rados.0.smithi105.stdout:681: finishing write tid 3 to smithi10548823-6 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:681: finishing write tid 4 to smithi10548823-6 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:681: finishing write tid 5 to smithi10548823-6 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:681: oid 6 updating version 0 to 31 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:681: oid 6 version 31 is already newer than 29 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:681: oid 6 updating version 31 to 32 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:681: oid 6 version 32 is already newer than 30 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 32 (ObjNum 80 snap 1 seq_num 80) dirty exists 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:681: left oid 6 (ObjNum 80 snap 1 seq_num 80) 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:684: got (0) Success 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:685: finishing write tid 1 to smithi10548823-1 2024-04-20T02:50:47.511 INFO:tasks.rados.rados.0.smithi105.stdout:688: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-04-20T02:50:47.516 INFO:teuthology.orchestra.run.smithi138.stdout:reshard success 2024-04-20T02:50:47.516 INFO:tasks.rados.rados.0.smithi105.stdout:688: writing smithi10548823-18 from 594981 to 1204178 tid 1 2024-04-20T02:50:47.525 INFO:tasks.rados.rados.0.smithi105.stdout:688: writing smithi10548823-18 from 1974123 to 2719151 tid 2 2024-04-20T02:50:47.529 INFO:tasks.rados.rados.0.smithi105.stdout:688: writing smithi10548823-18 from 3279873 to 3394981 tid 3 2024-04-20T02:50:47.529 INFO:tasks.rados.rados.0.smithi105.stdout:676: done (10 left) 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:679: done (9 left) 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:680: done (8 left) 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:681: done (7 left) 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:682: done (6 left) 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:683: done (5 left) 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:684: done (4 left) 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:689: read oid 36 snap -1 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:689: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:690: tier_promote oid 33 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:685: finishing write tid 2 to smithi10548823-1 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:685: finishing write tid 3 to smithi10548823-1 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:685: finishing write tid 4 to smithi10548823-1 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:685: finishing write tid 5 to smithi10548823-1 2024-04-20T02:50:47.530 INFO:tasks.rados.rados.0.smithi105.stdout:685: oid 1 updating version 0 to 53 2024-04-20T02:50:47.531 INFO:tasks.rados.rados.0.smithi105.stdout:685: oid 1 updating version 53 to 54 2024-04-20T02:50:47.531 INFO:tasks.rados.rados.0.smithi105.stdout:685: oid 1 updating version 54 to 56 2024-04-20T02:50:47.531 INFO:tasks.rados.rados.0.smithi105.stdout:685: oid 1 version 56 is already newer than 55 2024-04-20T02:50:47.531 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 56 (ObjNum 81 snap 1 seq_num 81) dirty exists 2024-04-20T02:50:47.531 INFO:tasks.rados.rados.0.smithi105.stdout:685: left oid 1 (ObjNum 81 snap 1 seq_num 81) 2024-04-20T02:50:47.531 INFO:tasks.rados.rados.0.smithi105.stdout:685: done (5 left) 2024-04-20T02:50:47.531 INFO:tasks.rados.rados.0.smithi105.stdout:691: snap_create 2024-04-20T02:50:47.531 INFO:tasks.rados.rados.0.smithi105.stdout:687: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-20T02:50:47.607 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:47.606+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:47.611 INFO:tasks.rados.rados.0.smithi105.stdout:686: finishing write tid 1 to smithi10548823-48 2024-04-20T02:50:47.611 INFO:tasks.rados.rados.0.smithi105.stdout:686: finishing write tid 2 to smithi10548823-48 2024-04-20T02:50:47.611 INFO:tasks.rados.rados.0.smithi105.stdout:686: finishing write tid 3 to smithi10548823-48 2024-04-20T02:50:47.611 INFO:tasks.rados.rados.0.smithi105.stdout:686: finishing write tid 4 to smithi10548823-48 2024-04-20T02:50:47.611 INFO:tasks.rados.rados.0.smithi105.stdout:686: finishing write tid 5 to smithi10548823-48 2024-04-20T02:50:47.611 INFO:tasks.rados.rados.0.smithi105.stdout:686: oid 48 updating version 0 to 25 2024-04-20T02:50:47.611 INFO:tasks.rados.rados.0.smithi105.stdout:686: oid 48 version 25 is already newer than 22 2024-04-20T02:50:47.612 INFO:tasks.rados.rados.0.smithi105.stdout:686: oid 48 version 25 is already newer than 23 2024-04-20T02:50:47.612 INFO:tasks.rados.rados.0.smithi105.stdout:686: oid 48 version 25 is already newer than 24 2024-04-20T02:50:47.612 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 25 (ObjNum 82 snap 1 seq_num 82) dirty exists 2024-04-20T02:50:47.612 INFO:tasks.rados.rados.0.smithi105.stdout:686: left oid 48 (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:50:47.612 INFO:tasks.rados.rados.0.smithi105.stdout:689: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-20T02:50:47.617 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:47.617 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:47.628 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:50:47.694 INFO:tasks.rados.rados.0.smithi105.stdout:688: finishing write tid 1 to smithi10548823-18 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:688: finishing write tid 2 to smithi10548823-18 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:688: finishing write tid 3 to smithi10548823-18 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:688: finishing write tid 4 to smithi10548823-18 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:690: got (0) Success 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 3 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:688: finishing write tid 5 to smithi10548823-18 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:688: oid 18 updating version 0 to 24 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:688: oid 18 updating version 24 to 26 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:688: oid 18 version 26 is already newer than 25 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:688: oid 18 version 26 is already newer than 23 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 26 (ObjNum 83 snap 1 seq_num 83) dirty exists 2024-04-20T02:50:47.695 INFO:tasks.rados.rados.0.smithi105.stdout:688: left oid 18 (ObjNum 83 snap 1 seq_num 83) 2024-04-20T02:50:47.712 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:47.712+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:47.798 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:47.798 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:47.809 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:50:47.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:47.809+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:47.910 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:47.911+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:47.930 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.5 2024-04-20T02:50:47.930 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-5 fsck 2024-04-20T02:50:47.986 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:47.987 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:47.998 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:50:48.011 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:48.011+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:48.111 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:48.111+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:48.126 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:48.126 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:48.136 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:50:48.221 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:48.221+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:48.313 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:48.313+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:48.314 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:48.314 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:48.327 INFO:tasks.rados.rados.0.smithi105.stdout:686: done (5 left) 2024-04-20T02:50:48.327 INFO:tasks.rados.rados.0.smithi105.stdout:687: done (4 left) 2024-04-20T02:50:48.327 INFO:tasks.rados.rados.0.smithi105.stdout:688: done (3 left) 2024-04-20T02:50:48.327 INFO:tasks.rados.rados.0.smithi105.stdout:689: done (2 left) 2024-04-20T02:50:48.327 INFO:tasks.rados.rados.0.smithi105.stdout:690: done (1 left) 2024-04-20T02:50:48.327 INFO:tasks.rados.rados.0.smithi105.stdout:691: done (0 left) 2024-04-20T02:50:48.327 INFO:tasks.rados.rados.0.smithi105.stdout:692: write (excl) oid 27 current snap is 2 2024-04-20T02:50:48.327 INFO:tasks.rados.rados.0.smithi105.stdout:692: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-04-20T02:50:48.330 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:50:48.332 INFO:tasks.rados.rados.0.smithi105.stdout:692: writing smithi10548823-27 from 611788 to 1012587 tid 1 2024-04-20T02:50:48.339 INFO:tasks.rados.rados.0.smithi105.stdout:692: writing smithi10548823-27 from 1748958 to 2237644 tid 2 2024-04-20T02:50:48.341 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:50:48.346 INFO:tasks.rados.rados.0.smithi105.stdout:692: writing smithi10548823-27 from 2907296 to 3411788 tid 3 2024-04-20T02:50:48.346 INFO:tasks.rados.rados.0.smithi105.stdout:693: set_chunk oid 4 target oid 2024-04-20T02:50:48.395 INFO:tasks.rados.rados.0.smithi105.stdout:693: set_chunk oid 4 offset: 1863168 length: 949760 target oid 674d5427330032ee1b577d2d4ea1afb19e8e888623da7eb444911905b8d2546f offset: 0 snap 1 2024-04-20T02:50:48.395 INFO:tasks.rados.rados.0.smithi105.stdout:694: rollback oid 33 current snap is 2 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 33 to 1 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: finishing write tid 1 to smithi10548823-27 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: finishing write tid 2 to smithi10548823-27 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: finishing write tid 3 to smithi10548823-27 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: finishing write tid 4 to smithi10548823-27 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: finishing write tid 5 to smithi10548823-27 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: oid 27 updating version 0 to 36 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: oid 27 version 36 is already newer than 34 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: oid 27 updating version 36 to 37 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: oid 27 version 37 is already newer than 35 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 37 (ObjNum 84 snap 2 seq_num 84) dirty exists 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: left oid 27 (ObjNum 84 snap 2 seq_num 84) 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:692: done (2 left) 2024-04-20T02:50:48.396 INFO:tasks.rados.rados.0.smithi105.stdout:695: rollback oid 47 current snap is 2 2024-04-20T02:50:48.397 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 47 to 1 2024-04-20T02:50:48.397 INFO:tasks.rados.rados.0.smithi105.stdout:696: tier_promote oid 7 2024-04-20T02:50:48.397 INFO:tasks.rados.rados.0.smithi105.stdout:697: write oid 29 current snap is 2 2024-04-20T02:50:48.397 INFO:tasks.rados.rados.0.smithi105.stdout:697: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-04-20T02:50:48.401 INFO:tasks.rados.rados.0.smithi105.stdout:697: writing smithi10548823-29 from 628595 to 1104643 tid 1 2024-04-20T02:50:48.413 INFO:tasks.rados.rados.0.smithi105.stdout:697: writing smithi10548823-29 from 1807440 to 2439784 tid 2 2024-04-20T02:50:48.414 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:48.414+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:48.415 INFO:tasks.rados.rados.0.smithi105.stdout:697: writing smithi10548823-29 from 2934719 to 3387193 tid 3 2024-04-20T02:50:48.415 INFO:tasks.rados.rados.0.smithi105.stdout:698: read oid 10 snap -1 2024-04-20T02:50:48.415 INFO:tasks.rados.rados.0.smithi105.stdout:698: expect (ObjNum 77 snap 1 seq_num 77) 2024-04-20T02:50:48.416 INFO:tasks.rados.rados.0.smithi105.stdout:699: write oid 8 current snap is 2 2024-04-20T02:50:48.416 INFO:tasks.rados.rados.0.smithi105.stdout:693: finishing set_chunk to oid 4 2024-04-20T02:50:48.416 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 1863168 length: 949760 2024-04-20T02:50:48.416 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 73 snap 1 seq_num 73) dirty exists 2024-04-20T02:50:48.416 INFO:tasks.rados.rados.0.smithi105.stdout:699: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-04-20T02:50:48.422 INFO:tasks.rados.rados.0.smithi105.stdout:699: writing smithi10548823-8 from 645402 to 1196699 tid 1 2024-04-20T02:50:48.430 INFO:tasks.rados.rados.0.smithi105.stdout:699: writing smithi10548823-8 from 1749569 to 2241924 tid 2 2024-04-20T02:50:48.438 INFO:tasks.rados.rados.0.smithi105.stdout:699: writing smithi10548823-8 from 2845789 to 3445402 tid 3 2024-04-20T02:50:48.438 INFO:tasks.rados.rados.0.smithi105.stdout:693: done (6 left) 2024-04-20T02:50:48.438 INFO:tasks.rados.rados.0.smithi105.stdout:700: read oid 4 snap 1 2024-04-20T02:50:48.438 INFO:tasks.rados.rados.0.smithi105.stdout:700: expect (ObjNum 73 snap 1 seq_num 73) 2024-04-20T02:50:48.438 INFO:tasks.rados.rados.0.smithi105.stdout:701: write (excl) oid 41 current snap is 2 2024-04-20T02:50:48.438 INFO:tasks.rados.rados.0.smithi105.stdout:694: finishing rollback tid 0 to smithi10548823-33 2024-04-20T02:50:48.438 INFO:tasks.rados.rados.0.smithi105.stdout:694: finishing rollback tid 1 to smithi10548823-33 2024-04-20T02:50:48.438 INFO:tasks.rados.rados.0.smithi105.stdout:701: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-04-20T02:50:48.445 INFO:tasks.rados.rados.0.smithi105.stdout:701: writing smithi10548823-41 from 662209 to 1288755 tid 1 2024-04-20T02:50:48.453 INFO:tasks.rados.rados.0.smithi105.stdout:701: writing smithi10548823-41 from 1808051 to 2444064 tid 2 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:701: writing smithi10548823-41 from 2873212 to 3462209 tid 3 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:702: read oid 32 snap -1 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:702: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:703: write oid 19 current snap is 2 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:694: finishing rollback tid 2 to smithi10548823-33 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 17 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:695: finishing rollback tid 0 to smithi10548823-47 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:695: finishing rollback tid 1 to smithi10548823-47 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:695: finishing rollback tid 2 to smithi10548823-47 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 24 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:696: got (0) Success 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 9 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-20T02:50:48.460 INFO:tasks.rados.rados.0.smithi105.stdout:698: expect (ObjNum 77 snap 1 seq_num 77) 2024-04-20T02:50:48.514 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:48.514+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:48.528 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:48.529 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:48.539 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:50:48.623 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:48.623+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:48.649 INFO:tasks.rados.rados.0.smithi105.stdout:697: finishing write tid 1 to smithi10548823-29 2024-04-20T02:50:48.650 INFO:tasks.rados.rados.0.smithi105.stdout:700: expect (ObjNum 73 snap 1 seq_num 73) 2024-04-20T02:50:48.707 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:48.708 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:48.716 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:48.716+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:48.718 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:50:48.747 INFO:teuthology.orchestra.run.smithi138.stdout:fsck success 2024-04-20T02:50:48.755 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-04-20T02:50:48.755 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-04-20T02:50:48.755 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-20T02:50:48.755 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-20T02:50:48.798 INFO:tasks.ceph.osd.5:Started 2024-04-20T02:50:48.798 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:50:48.816 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:48.817+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:48.837 INFO:tasks.rados.rados.0.smithi105.stdout:697: finishing write tid 2 to smithi10548823-29 2024-04-20T02:50:48.837 INFO:tasks.rados.rados.0.smithi105.stdout:697: finishing write tid 3 to smithi10548823-29 2024-04-20T02:50:48.837 INFO:tasks.rados.rados.0.smithi105.stdout:703: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-04-20T02:50:48.845 INFO:tasks.rados.rados.0.smithi105.stdout:703: writing smithi10548823-19 from 679016 to 1380811 tid 1 2024-04-20T02:50:48.847 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:48.847 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:48.850 INFO:tasks.rados.rados.0.smithi105.stdout:703: writing smithi10548823-19 from 1866533 to 2362557 tid 2 2024-04-20T02:50:48.855 INFO:tasks.rados.rados.0.smithi105.stdout:703: writing smithi10548823-19 from 2900635 to 3303278 tid 3 2024-04-20T02:50:48.856 INFO:tasks.rados.rados.0.smithi105.stdout:694: done (9 left) 2024-04-20T02:50:48.856 INFO:tasks.rados.rados.0.smithi105.stdout:695: done (8 left) 2024-04-20T02:50:48.856 INFO:tasks.rados.rados.0.smithi105.stdout:696: done (7 left) 2024-04-20T02:50:48.856 INFO:tasks.rados.rados.0.smithi105.stdout:698: done (6 left) 2024-04-20T02:50:48.856 INFO:tasks.rados.rados.0.smithi105.stdout:700: done (5 left) 2024-04-20T02:50:48.856 INFO:tasks.rados.rados.0.smithi105.stdout:704: snap_remove snap 1 2024-04-20T02:50:48.858 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:50:48.918 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:48.917+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:48.931 INFO:teuthology.orchestra.run.smithi138.stderr:no valid command found; 10 closest matches: 2024-04-20T02:50:48.931 INFO:teuthology.orchestra.run.smithi138.stderr:0 2024-04-20T02:50:48.931 INFO:teuthology.orchestra.run.smithi138.stderr:1 2024-04-20T02:50:48.932 INFO:teuthology.orchestra.run.smithi138.stderr:2 2024-04-20T02:50:48.932 INFO:teuthology.orchestra.run.smithi138.stderr:abort 2024-04-20T02:50:48.932 INFO:teuthology.orchestra.run.smithi138.stderr:assert 2024-04-20T02:50:48.932 INFO:teuthology.orchestra.run.smithi138.stderr:config diff 2024-04-20T02:50:48.932 INFO:teuthology.orchestra.run.smithi138.stderr:config diff get 2024-04-20T02:50:48.932 INFO:teuthology.orchestra.run.smithi138.stderr:config get 2024-04-20T02:50:48.932 INFO:teuthology.orchestra.run.smithi138.stderr:config help [] 2024-04-20T02:50:48.932 INFO:teuthology.orchestra.run.smithi138.stderr:config set ... 2024-04-20T02:50:48.932 INFO:teuthology.orchestra.run.smithi138.stderr:admin_socket: invalid command 2024-04-20T02:50:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T02:50:48.934 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-04-20T02:50:49.019 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:49.019+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:49.026 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:49.026 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:49.036 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:50:49.119 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:49.119+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:49.205 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:49.205 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:49.216 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:50:49.260 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:49.260+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:49.321 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:49.321+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:49.338 INFO:tasks.rados.rados.0.smithi105.stdout:704: done (5 left) 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:705: rollback oid 2 current snap is 2 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:705: done (5 left) 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:706: read oid 24 snap -1 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:706: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:707: read oid 17 snap -1 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:707: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:708: read oid 20 snap -1 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:708: expect deleted 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:709: read oid 21 snap -1 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:709: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:710: tier_promote oid 28 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:697: finishing write tid 4 to smithi10548823-29 2024-04-20T02:50:49.339 INFO:tasks.rados.rados.0.smithi105.stdout:697: finishing write tid 5 to smithi10548823-29 2024-04-20T02:50:49.340 INFO:tasks.rados.rados.0.smithi105.stdout:697: oid 29 updating version 0 to 13 2024-04-20T02:50:49.340 INFO:tasks.rados.rados.0.smithi105.stdout:697: oid 29 version 13 is already newer than 11 2024-04-20T02:50:49.340 INFO:tasks.rados.rados.0.smithi105.stdout:697: oid 29 version 13 is already newer than 12 2024-04-20T02:50:49.340 INFO:tasks.rados.rados.0.smithi105.stdout:697: oid 29 version 13 is already newer than 10 2024-04-20T02:50:49.340 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 13 (ObjNum 85 snap 2 seq_num 85) dirty exists 2024-04-20T02:50:49.340 INFO:tasks.rados.rados.0.smithi105.stdout:697: left oid 29 (ObjNum 85 snap 2 seq_num 85) 2024-04-20T02:50:49.340 INFO:tasks.rados.rados.0.smithi105.stdout:697: done (9 left) 2024-04-20T02:50:49.340 INFO:tasks.rados.rados.0.smithi105.stdout:711: set_chunk oid 46 target oid 2024-04-20T02:50:49.350 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:50:49.357 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:49.358 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:49.373 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:50:49.392 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:49.392+0000 7fcba0c807c0 -1 Falling back to public interface 2024-04-20T02:50:49.405 INFO:tasks.rados.rados.0.smithi105.stdout:711: set_chunk oid 46 offset: 462848 length: 2266624 target oid 7743decba76c63dac84fe69ef795bba58316ab29326b9e251ea9f78a44150983 offset: 0 snap -1 2024-04-20T02:50:49.405 INFO:tasks.rados.rados.0.smithi105.stdout:712: write (excl) oid 31 current snap is 2 2024-04-20T02:50:49.405 INFO:tasks.rados.rados.0.smithi105.stdout:712: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-04-20T02:50:49.412 INFO:tasks.rados.rados.0.smithi105.stdout:712: writing smithi10548823-31 from 695823 to 1472867 tid 1 2024-04-20T02:50:49.418 INFO:tasks.rados.rados.0.smithi105.stdout:712: writing smithi10548823-31 from 1925015 to 2564697 tid 2 2024-04-20T02:50:49.421 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:49.422+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:49.424 INFO:tasks.rados.rados.0.smithi105.stdout:712: writing smithi10548823-31 from 3328058 to 3495823 tid 3 2024-04-20T02:50:49.424 INFO:tasks.rados.rados.0.smithi105.stdout:713: read oid 36 snap -1 2024-04-20T02:50:49.424 INFO:tasks.rados.rados.0.smithi105.stdout:713: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-20T02:50:49.424 INFO:tasks.rados.rados.0.smithi105.stdout:714: set_chunk oid 13 target oid 2024-04-20T02:50:49.443 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:50:49.481 INFO:tasks.rados.rados.0.smithi105.stdout:714: set_chunk oid 13 offset: 319488 length: 2148864 target oid a7c07b0681810927cd236a7b8682ee70c376fc25d148820e2f990e620fcf9282 offset: 0 snap -1 2024-04-20T02:50:49.481 INFO:tasks.rados.rados.0.smithi105.stdout:715: read oid 23 snap -1 2024-04-20T02:50:49.482 INFO:tasks.rados.rados.0.smithi105.stdout:715: expect (ObjNum 67 snap 0 seq_num 67) 2024-04-20T02:50:49.482 INFO:tasks.rados.rados.0.smithi105.stdout:716: rollback oid 26 current snap is 2 2024-04-20T02:50:49.482 INFO:tasks.rados.rados.0.smithi105.stdout:716: done (14 left) 2024-04-20T02:50:49.482 INFO:tasks.rados.rados.0.smithi105.stdout:717: read oid 42 snap -1 2024-04-20T02:50:49.482 INFO:tasks.rados.rados.0.smithi105.stdout:717: expect (ObjNum 41 snap 0 seq_num 41) 2024-04-20T02:50:49.482 INFO:tasks.rados.rados.0.smithi105.stdout:718: delete oid 50 current snap is 2 2024-04-20T02:50:49.482 INFO:tasks.rados.rados.0.smithi105.stdout:702: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-20T02:50:49.522 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:49.522+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:49.549 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:49.549 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:49.560 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:50:49.577 INFO:tasks.rados.rados.0.smithi105.stdout:699: finishing write tid 1 to smithi10548823-8 2024-04-20T02:50:49.577 INFO:tasks.rados.rados.0.smithi105.stdout:699: finishing write tid 2 to smithi10548823-8 2024-04-20T02:50:49.577 INFO:tasks.rados.rados.0.smithi105.stdout:702: done (15 left) 2024-04-20T02:50:49.577 INFO:tasks.rados.rados.0.smithi105.stdout:718: done (14 left) 2024-04-20T02:50:49.577 INFO:tasks.rados.rados.0.smithi105.stdout:719: write oid 37 current snap is 2 2024-04-20T02:50:49.577 INFO:tasks.rados.rados.0.smithi105.stdout:699: finishing write tid 3 to smithi10548823-8 2024-04-20T02:50:49.578 INFO:tasks.rados.rados.0.smithi105.stdout:719: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-04-20T02:50:49.586 INFO:tasks.rados.rados.0.smithi105.stdout:719: writing smithi10548823-37 from 729437 to 1256979 tid 1 2024-04-20T02:50:49.592 INFO:tasks.rados.rados.0.smithi105.stdout:719: writing smithi10548823-37 from 1925626 to 2568977 tid 2 2024-04-20T02:50:49.598 INFO:tasks.rados.rados.0.smithi105.stdout:719: writing smithi10548823-37 from 3266551 to 3529437 tid 3 2024-04-20T02:50:49.598 INFO:tasks.rados.rados.0.smithi105.stdout:720: read oid 6 snap -1 2024-04-20T02:50:49.598 INFO:tasks.rados.rados.0.smithi105.stdout:720: expect (ObjNum 80 snap 1 seq_num 80) 2024-04-20T02:50:49.598 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:49.598 INFO:tasks.rados.rados.0.smithi105.stdout:699: finishing write tid 4 to smithi10548823-8 2024-04-20T02:50:49.598 INFO:tasks.rados.rados.0.smithi105.stdout:699: finishing write tid 5 to smithi10548823-8 2024-04-20T02:50:49.598 INFO:tasks.rados.rados.0.smithi105.stdout:699: oid 8 updating version 0 to 28 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:699: oid 8 version 28 is already newer than 27 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:699: oid 8 version 28 is already newer than 26 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:699: oid 8 updating version 28 to 29 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 29 (ObjNum 86 snap 2 seq_num 86) dirty exists 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:699: left oid 8 (ObjNum 86 snap 2 seq_num 86) 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:701: finishing write tid 1 to smithi10548823-41 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:701: finishing write tid 2 to smithi10548823-41 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:701: finishing write tid 3 to smithi10548823-41 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:701: finishing write tid 4 to smithi10548823-41 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:701: finishing write tid 5 to smithi10548823-41 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:701: oid 41 updating version 0 to 62 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:701: oid 41 version 62 is already newer than 60 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:701: oid 41 updating version 62 to 63 2024-04-20T02:50:49.599 INFO:tasks.rados.rados.0.smithi105.stdout:701: oid 41 version 63 is already newer than 61 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 63 (ObjNum 87 snap 2 seq_num 87) dirty exists 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:701: left oid 41 (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:703: finishing write tid 1 to smithi10548823-19 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:703: finishing write tid 2 to smithi10548823-19 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:703: finishing write tid 3 to smithi10548823-19 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:703: finishing write tid 4 to smithi10548823-19 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:703: finishing write tid 5 to smithi10548823-19 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:703: oid 19 updating version 0 to 45 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:703: oid 19 updating version 45 to 47 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:703: oid 19 version 47 is already newer than 44 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:703: oid 19 version 47 is already newer than 46 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 47 (ObjNum 88 snap 2 seq_num 88) dirty exists 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:703: left oid 19 (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:50:49.600 INFO:tasks.rados.rados.0.smithi105.stdout:709: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-20T02:50:49.624 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:49.624+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:49.631 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:50:49.651 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:50:49.670 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:50:49.682 INFO:tasks.rados.rados.0.smithi105.stdout:706: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-20T02:50:49.690 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:50:49.710 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:50:49.719 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:49.719 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:50:49.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:50:49.719 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:50:49.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.719 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:926 4.3 e32/27 4:cf11ccc0:::smithi10548823-4:head v 32'46, mlcod=30'43)", 2024-04-20T02:50:49.719 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.383063+0000", 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.3335785819999999, 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017923799000000001, 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.383063+0000", 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.383063+0000", 2024-04-20T02:50:49.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.383062+0000", 2024-04-20T02:50:49.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:49.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.383468+0000", 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040624999999999998 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.383486+0000", 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7640000000000001e-05 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.383495+0000", 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5880000000000004e-06 2024-04-20T02:50:49.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.383550+0000", 2024-04-20T02:50:49.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4459999999999997e-05 2024-04-20T02:50:49.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.383574+0000", 2024-04-20T02:50:49.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4820999999999998e-05 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400943+0000", 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017368403000000001 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400987+0000", 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3902000000000001e-05 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.725 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:927 4.3 e32/27 4:cf11ccc0:::smithi10548823-4:head v 32'47, mlcod=30'43)", 2024-04-20T02:50:49.725 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.408078+0000", 2024-04-20T02:50:49.725 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.3085639059999998, 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018319218000000002, 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.408078+0000", 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.408078+0000", 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.408077+0000", 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.408141+0000", 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4430000000000005e-05 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.408155+0000", 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4397e-05 2024-04-20T02:50:49.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.408163+0000", 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4719999999999998e-06 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.408210+0000", 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6718000000000002e-05 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.408235+0000", 2024-04-20T02:50:49.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4882e-05 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426335+0000", 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018100013000000002 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426397+0000", 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2273000000000005e-05 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:936 4.3 e32/27 4:ca036f62:::smithi10548823-34:head v 32'49, mlcod=32'45)", 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.412413+0000", 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.3042288339999999, 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028737441999999998, 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412413+0000", 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412413+0000", 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412412+0000", 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412534+0000", 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012218799999999999 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412549+0000", 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4674999999999999e-05 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412556+0000", 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.165e-06 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412597+0000", 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0491999999999998e-05 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412619+0000", 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2776999999999999e-05 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.441082+0000", 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028462271000000001 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.441150+0000", 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8527000000000003e-05 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.733 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:937 4.3 e32/27 4:ca036f62:::smithi10548823-34:head v 32'50, mlcod=32'45)", 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.426221+0000", 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.2904203650000001, 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033574451999999998, 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426221+0000", 2024-04-20T02:50:49.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426221+0000", 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426220+0000", 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426493+0000", 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027307600000000003 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426507+0000", 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3642000000000001e-05 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426515+0000", 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4689999999999997e-06 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426551+0000", 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5926000000000001e-05 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426572+0000", 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0732000000000001e-05 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.459759+0000", 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033187333999999999 2024-04-20T02:50:49.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.459796+0000", 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6451e-05 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:938 4.3 e32/27 4:ca036f62:::smithi10548823-34:head v 32'51, mlcod=32'47)", 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.437179+0000", 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.279462278, 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023687592, 2024-04-20T02:50:49.738 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437179+0000", 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437179+0000", 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437178+0000", 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437438+0000", 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025980699999999997 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437449+0000", 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1219999999999999e-05 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437455+0000", 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.852e-06 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.441180+0000", 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037249169999999999 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.441203+0000", 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3071e-05 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460839+0000", 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019635940000000001 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460867+0000", 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7616000000000001e-05 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.1315010910000001, 2024-04-20T02:50:49.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.1154828129999999, 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.748 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:50:49.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:50:49.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:50:49.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:50:49.753 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.1153295839999999, 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:49.754 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.1152195749999998, 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:49.760 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:50:49.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:50:49.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:50:49.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:50:49.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.115189199, 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:49.766 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.770 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.1107717799999999, 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.771 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:49.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:50:49.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:50:49.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.101520029, 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.0989978859999998, 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:50:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:50:49.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:50:49.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:50:49.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:50:49.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:50:49.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:50:49.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.0989130189999998, 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.790 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:50:49.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:50:49.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:50:49.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.0988954089999998, 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:49.796 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.800 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:991 4.b e32/22 4:d4308fef:::smithi10548823-6:head v 32'29, mlcod=30'27)", 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.991991155, 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046400134000000003, 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724649+0000", 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724907+0000", 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025779699999999998 2024-04-20T02:50:49.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724924+0000", 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7110000000000001e-05 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724934+0000", 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5370000000000003e-06 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724984+0000", 2024-04-20T02:50:49.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0791000000000002e-05 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.725011+0000", 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6299e-05 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771009+0000", 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045998144999999997 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771050+0000", 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1727999999999998e-05 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:992 4.b e32/22 4:d4308fef:::smithi10548823-6:head v 32'30, mlcod=30'27)", 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.742718+0000", 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.9739230189999999, 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039829634000000003, 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742718+0000", 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742718+0000", 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742718+0000", 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:49.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742891+0000", 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000173296 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742903+0000", 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2138e-05 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742911+0000", 2024-04-20T02:50:49.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0350000000000001e-06 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.748598+0000", 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056863449999999998 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.748626+0000", 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8853999999999999e-05 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.782519+0000", 2024-04-20T02:50:49.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033892275999999999 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.782548+0000", 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9473000000000001e-05 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.3.13:40 3.0 3:1bb16482:::5f0a80f25682b404598836d7b959970cfcfaa9e1d8173bf2c8978ef66442ee5f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.751071+0000", 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.9655708189999999, 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024177230000000001, 2024-04-20T02:50:49.809 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751071+0000", 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751071+0000", 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751070+0000", 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751103+0000", 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3308e-05 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751110+0000", 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4609999999999996e-06 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751117+0000", 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1570000000000003e-06 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.762618+0000", 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01150057 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.762712+0000", 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4554000000000003e-05 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.762924+0000", 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021163900000000001 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.767079+0000", 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0041550490000000001 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.769619+0000", 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025399950000000002 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.775124+0000", 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0055048980000000003 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.775183+0000", 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8998e-05 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.775248+0000", 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5166000000000006e-05 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.814 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1003 4.3 e32/27 4:cd374330:::smithi10548823-1:head v 32'53, mlcod=32'51)", 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:47.240324+0000", 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.4763179279999998, 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028418235999999999, 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.240324+0000", 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.240324+0000", 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.240322+0000", 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.240453+0000", 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000131075 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.240470+0000", 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7368999999999999e-05 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.240478+0000", 2024-04-20T02:50:49.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.418e-06 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.240533+0000", 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4854e-05 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.240559+0000", 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6644000000000001e-05 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.268690+0000", 2024-04-20T02:50:49.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028130542000000001 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.268742+0000", 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2030000000000002e-05 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1004 4.3 e32/27 4:cd374330:::smithi10548823-1:head v 32'54, mlcod=32'51)", 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:47.251137+0000", 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.4655047400000001, 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018283840999999999, 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:49.819 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.251137+0000", 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.251137+0000", 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.251135+0000", 2024-04-20T02:50:49.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.251299+0000", 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000163258 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.251313+0000", 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3997e-05 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.251321+0000", 2024-04-20T02:50:49.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.208e-06 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.251368+0000", 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7012999999999999e-05 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.251395+0000", 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7086000000000001e-05 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.269395+0000", 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01800005 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:47.269421+0000", 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5646000000000001e-05 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:49.823 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:49.825 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:50:49.825 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:50:49.825 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:50:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:50:49.826 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:50:49.826 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:50:49.827 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:50:49.827 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:50:49.827 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:50:49.827 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-20T02:50:49.827 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:50:49.827 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:50:49.827 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:50:49.827 INFO:tasks.rados.rados.0.smithi105.stdout:707: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-20T02:50:49.828 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:49.825+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:49.882 INFO:tasks.rados.rados.0.smithi105.stdout:710: got (0) Success 2024-04-20T02:50:49.882 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:49.882 INFO:tasks.rados.rados.0.smithi105.stdout:711: finishing set_chunk to oid 46 2024-04-20T02:50:49.882 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 46 set_chunk returned error code -95 offset: 462848 length: 2266624 2024-04-20T02:50:49.882 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-20T02:50:49.882 INFO:tasks.rados.rados.0.smithi105.stdout:713: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-20T02:50:49.925 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:49.925+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:49.970 INFO:tasks.rados.rados.0.smithi105.stdout:712: finishing write tid 1 to smithi10548823-31 2024-04-20T02:50:49.970 INFO:tasks.rados.rados.0.smithi105.stdout:712: finishing write tid 2 to smithi10548823-31 2024-04-20T02:50:49.970 INFO:tasks.rados.rados.0.smithi105.stdout:699: done (15 left) 2024-04-20T02:50:49.970 INFO:tasks.rados.rados.0.smithi105.stdout:701: done (14 left) 2024-04-20T02:50:49.970 INFO:tasks.rados.rados.0.smithi105.stdout:703: done (13 left) 2024-04-20T02:50:49.970 INFO:tasks.rados.rados.0.smithi105.stdout:706: done (12 left) 2024-04-20T02:50:49.970 INFO:tasks.rados.rados.0.smithi105.stdout:707: done (11 left) 2024-04-20T02:50:49.970 INFO:tasks.rados.rados.0.smithi105.stdout:708: done (10 left) 2024-04-20T02:50:49.970 INFO:tasks.rados.rados.0.smithi105.stdout:709: done (9 left) 2024-04-20T02:50:49.970 INFO:tasks.rados.rados.0.smithi105.stdout:710: done (8 left) 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:711: done (7 left) 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:713: done (6 left) 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:721: read oid 47 snap -1 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:721: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:722: read oid 48 snap -1 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:722: expect (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:723: read oid 39 snap -1 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:723: expect (ObjNum 76 snap 1 seq_num 76) 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:724: read oid 16 snap -1 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:724: expect deleted 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:725: snap_create 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:712: finishing write tid 3 to smithi10548823-31 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:712: finishing write tid 4 to smithi10548823-31 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:712: finishing write tid 5 to smithi10548823-31 2024-04-20T02:50:49.971 INFO:tasks.rados.rados.0.smithi105.stdout:712: oid 31 updating version 0 to 24 2024-04-20T02:50:49.972 INFO:tasks.rados.rados.0.smithi105.stdout:712: oid 31 updating version 24 to 26 2024-04-20T02:50:49.972 INFO:tasks.rados.rados.0.smithi105.stdout:712: oid 31 updating version 26 to 27 2024-04-20T02:50:49.972 INFO:tasks.rados.rados.0.smithi105.stdout:712: oid 31 version 27 is already newer than 25 2024-04-20T02:50:49.972 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 27 (ObjNum 89 snap 2 seq_num 89) dirty exists 2024-04-20T02:50:49.972 INFO:tasks.rados.rados.0.smithi105.stdout:712: left oid 31 (ObjNum 89 snap 2 seq_num 89) 2024-04-20T02:50:49.972 INFO:tasks.rados.rados.0.smithi105.stdout:714: finishing set_chunk to oid 13 2024-04-20T02:50:49.972 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 46 (ObjNum 61 snap 0 seq_num 61) dirty exists 2024-04-20T02:50:49.972 INFO:tasks.rados.rados.0.smithi105.stdout:715: expect (ObjNum 67 snap 0 seq_num 67) 2024-04-20T02:50:49.999 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:49.999 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:49.999 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:50:49.999 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:50.009 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:50:50.051 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:50.051+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:50.120 INFO:tasks.rados.rados.0.smithi105.stdout:717: expect (ObjNum 41 snap 0 seq_num 41) 2024-04-20T02:50:50.126 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:50.127+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:50.179 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:50.180 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:50.180 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:50:50.180 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:50:50.180 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:50.190 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:50:50.211 INFO:tasks.rados.rados.0.smithi105.stdout:720: expect (ObjNum 80 snap 1 seq_num 80) 2024-04-20T02:50:50.255 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:50.255+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:50.328 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:50.328+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:50.343 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:50.343 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:50:50.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:50:50.343 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:50:50.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.343 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.0.15:83 3.0 e32/21 3:02c2f6cb:::d1f4d85977b2983a02dfb9155c67d2c8e89194bc9140d8a0abab1eabbc7b49fa:head v 32'53, mlcod=30'51)", 2024-04-20T02:50:50.343 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.411933+0000", 2024-04-20T02:50:50.343 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.931253543, 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0023930610000000001, 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411933+0000", 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411933+0000", 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411933+0000", 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411938+0000", 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3319999999999999e-06 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411954+0000", 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5502000000000002e-05 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411963+0000", 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9299999999999992e-06 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412806+0000", 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00084367399999999999 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412829+0000", 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2779e-05 2024-04-20T02:50:50.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.414293+0000", 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001463696 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.414326+0000", 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3488999999999997e-05 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.0.15:87 3.0 e32/21 3:37abdcb7:::smithi10548823-34 low_tier:head v 32'54, mlcod=30'52)", 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.455939+0000", 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.887247399, 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00078036900000000005, 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455939+0000", 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455939+0000", 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455938+0000", 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455949+0000", 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0447e-05 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455963+0000", 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4491999999999999e-05 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455981+0000", 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7575e-05 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456038+0000", 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7158999999999999e-05 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456062+0000", 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3960000000000001e-05 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456688+0000", 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00062628499999999997 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456720+0000", 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1418000000000003e-05 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.351 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.0.15:88 3.0 e32/21 3:37abdcb7:::smithi10548823-34 low_tier:head v 32'55, mlcod=32'53)", 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.460415+0000", 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.882771822, 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00085974499999999998, 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460415+0000", 2024-04-20T02:50:50.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460415+0000", 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460414+0000", 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460422+0000", 2024-04-20T02:50:50.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8820000000000001e-06 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460436+0000", 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4049999999999999e-05 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460444+0000", 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9189999999999999e-06 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460488+0000", 2024-04-20T02:50:50.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.392e-05 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460511+0000", 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2869000000000001e-05 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461244+0000", 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073307699999999995 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461275+0000", 2024-04-20T02:50:50.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1118000000000003e-05 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:950 3.0 e32/21 3:2249af83:::126d50b9ccd058128accdb5f9e4984ecfbb65aa7997f1616d3dcd0c09fefc945:head v 32'56, mlcod=32'54)", 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.530245+0000", 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.8129416759999999, 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00358348, 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.530245+0000", 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.530245+0000", 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.530244+0000", 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.530313+0000", 2024-04-20T02:50:50.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8801999999999996e-05 2024-04-20T02:50:50.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.530328+0000", 2024-04-20T02:50:50.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4833e-05 2024-04-20T02:50:50.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.530337+0000", 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9020000000000005e-06 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.530385+0000", 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8699999999999998e-05 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.530413+0000", 2024-04-20T02:50:50.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7827999999999999e-05 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.533792+0000", 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033789829999999999 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.533829+0000", 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6520999999999998e-05 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.1.13:23 3.0 e32/21 3:193d5116:::smithi10548823-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 32'57, mlcod=32'55)", 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.677216+0000", 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.6659712559999997, 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00081947300000000003, 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677216+0000", 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677216+0000", 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677214+0000", 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677225+0000", 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0848e-05 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677242+0000", 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7031999999999999e-05 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677252+0000", 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7010000000000008e-06 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677293+0000", 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1693999999999998e-05 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677312+0000", 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8765000000000001e-05 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677995+0000", 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068270699999999995 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678035+0000", 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0138000000000001e-05 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.364 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1000 3.0 e32/21 3:1ab8a438:::d1768af9860251b5b7bc5c43b5368761bba981c92dd8005238016b83940d5bad:head v 32'58, mlcod=32'56)", 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.740696+0000", 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.6024906460000001, 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00095835400000000002, 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.740696+0000", 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.740696+0000", 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.740694+0000", 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.740950+0000", 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000255274 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.740965+0000", 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5525000000000001e-05 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.740974+0000", 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2180000000000005e-06 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.741029+0000", 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4595000000000002e-05 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.741059+0000", 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9471e-05 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.741626+0000", 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00056742100000000003 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.741655+0000", 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8622000000000001e-05 2024-04-20T02:50:50.368 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.3.13:31 3.0 e32/21 3:151adb37:::837950dc06992f8eddc63ac63fa371572c3924ba8587b83653eec19f1bc0ad1c:head v 32'59, mlcod=32'57)", 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.751535+0000", 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.5916521189999999, 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010261421999999999, 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751535+0000", 2024-04-20T02:50:50.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751535+0000", 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751533+0000", 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751692+0000", 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000159061 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751713+0000", 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0360000000000002e-05 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751723+0000", 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7880000000000001e-06 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751786+0000", 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3132999999999993e-05 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751811+0000", 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5057000000000001e-05 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.761760+0000", 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0099489160000000004 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.761796+0000", 2024-04-20T02:50:50.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6440999999999999e-05 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.3.13:35 3.0 e32/21 3:141f2c22:::d3ef4b1d2c6a9efe723454f96663e0e6dc925fe46a0cd8dbc576dd437577011f:head v 32'60, mlcod=32'57)", 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.754608+0000", 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.5885787260000002, 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0072083060000000003, 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.754608+0000", 2024-04-20T02:50:50.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.754608+0000", 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.754607+0000", 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.754683+0000", 2024-04-20T02:50:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6061000000000002e-05 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.754693+0000", 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1239999999999994e-06 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.754704+0000", 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1024e-05 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.754743+0000", 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9656000000000001e-05 2024-04-20T02:50:50.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.754765+0000", 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1248000000000001e-05 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.761799+0000", 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0070349000000000002 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.761816+0000", 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7003e-05 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.3.13:36 3.0 e32/21 3:2492dc3d:::b4b703d984a4d4c0646840816f48a24181cf6c812ec9f072604ba6a73928dca1:head v 32'61, mlcod=32'57)", 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.758997+0000", 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.5841893709999999, 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033521879999999999, 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.758997+0000", 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.758997+0000", 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.758997+0000", 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.759255+0000", 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025812600000000002 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.759277+0000", 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1912e-05 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.759285+0000", 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5949999999999999e-06 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.759367+0000", 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1260999999999993e-05 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.759394+0000", 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7744999999999999e-05 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.762323+0000", 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002929114 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.762350+0000", 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6268999999999999e-05 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.380 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.3.13:40 3.0 e32/21 3:1bb16482:::5f0a80f25682b404598836d7b959970cfcfaa9e1d8173bf2c8978ef66442ee5f:head v 32'62, mlcod=32'57)", 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.765330+0000", 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.577857179, 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0041840640000000004, 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.765330+0000", 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.765330+0000", 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.765329+0000", 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.765365+0000", 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6477999999999999e-05 2024-04-20T02:50:50.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.765381+0000", 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5622000000000001e-05 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.765390+0000", 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9679999999999995e-06 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.765434+0000", 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3971999999999998e-05 2024-04-20T02:50:50.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.765458+0000", 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4031000000000001e-05 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.769476+0000", 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004017853 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.769514+0000", 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8089000000000001e-05 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.384 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.385 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.385 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.385 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.385 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:50.386 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:50:50.411 INFO:tasks.rados.rados.0.smithi105.stdout:719: finishing write tid 1 to smithi10548823-37 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:719: finishing write tid 2 to smithi10548823-37 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:719: finishing write tid 3 to smithi10548823-37 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:719: finishing write tid 4 to smithi10548823-37 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:719: finishing write tid 5 to smithi10548823-37 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:719: oid 37 updating version 0 to 19 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:719: oid 37 updating version 19 to 22 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:719: oid 37 version 22 is already newer than 20 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:719: oid 37 version 22 is already newer than 21 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 22 (ObjNum 91 snap 2 seq_num 91) dirty exists 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:719: left oid 37 (ObjNum 91 snap 2 seq_num 91) 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:712: done (10 left) 2024-04-20T02:50:50.412 INFO:tasks.rados.rados.0.smithi105.stdout:714: done (9 left) 2024-04-20T02:50:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:715: done (8 left) 2024-04-20T02:50:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:717: done (7 left) 2024-04-20T02:50:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:719: done (6 left) 2024-04-20T02:50:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:720: done (5 left) 2024-04-20T02:50:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:724: done (4 left) 2024-04-20T02:50:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:725: done (3 left) 2024-04-20T02:50:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:726: read oid 5 snap 2 2024-04-20T02:50:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:726: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-20T02:50:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:727: write oid 6 current snap is 3 2024-04-20T02:50:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:722: expect (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:50:50.455 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:50.455+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:50.530 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:50.530+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:50.541 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:50.541 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:50.542 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:50:50.542 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:50.552 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:50:50.563 INFO:tasks.rados.rados.0.smithi105.stdout:721: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:50:50.631 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:50.632+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:50.658 INFO:tasks.rados.rados.0.smithi105.stdout:723: expect (ObjNum 76 snap 1 seq_num 76) 2024-04-20T02:50:50.693 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:50.693 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:50.693 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:50:50.693 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:50:50.694 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:50.704 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:50:50.733 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:50.732+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:50.833 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:50.833+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:726: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-20T02:50:50.875 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:50.875 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:50:50.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:50:50.875 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:50:50.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.875 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:913 3.2 3:6ff86bb8:::5520356cbad01cbb77690f4acae83315abf571d3bc9027883d585e07c7f7b8e1:head [write 0~1270272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-20T02:50:50.875 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:45.323476+0000", 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.5510830369999997, 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018063643000000001, 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 913 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.323476+0000", 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.323476+0000", 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.323474+0000", 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.323982+0000", 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00050864499999999995 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.323997+0000", 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4959e-05 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.324005+0000", 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1419999999999993e-06 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.324060+0000", 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4367000000000003e-05 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.324171+0000", 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111019 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.324275+0000", 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104088 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.337406+0000", 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013131230000000001 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.339254+0000", 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0018480669999999999 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.341469+0000", 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00221456 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.341498+0000", 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9437e-05 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:45.341539+0000", 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1211999999999998e-05 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:941 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'19, mlcod=30'17)", 2024-04-20T02:50:50.881 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.3562676260000002, 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042482077, 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518290+0000", 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518988+0000", 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069878099999999999 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519003+0000", 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4756999999999999e-05 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519013+0000", 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4250000000000004e-06 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519071+0000", 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8628000000000002e-05 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519106+0000", 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4817000000000002e-05 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560724+0000", 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041618334 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560773+0000", 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8937999999999998e-05 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.885 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:942 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'20, mlcod=30'17)", 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.3374505169999997, 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030979784999999999, 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:50.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537107+0000", 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538354+0000", 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001247232 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538381+0000", 2024-04-20T02:50:50.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7008999999999998e-05 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538395+0000", 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4093999999999999e-05 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538476+0000", 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1045999999999996e-05 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538505+0000", 2024-04-20T02:50:50.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8969999999999999e-05 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568053+0000", 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029547370999999999 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568088+0000", 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5540000000000002e-05 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.3308192910000001, 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:50.895 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.3245982170000001, 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:50.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:50:50.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:50:50.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:943 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'21, mlcod=30'17)", 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.553675+0000", 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.3208841800000002, 2024-04-20T02:50:50.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014930229, 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553675+0000", 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553675+0000", 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553673+0000", 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553951+0000", 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027789900000000002 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553962+0000", 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0855e-05 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.553970+0000", 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1470000000000001e-06 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560816+0000", 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068456460000000004 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560851+0000", 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4555999999999999e-05 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568569+0000", 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.007718478 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568605+0000", 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5839e-05 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.3161980130000002, 2024-04-20T02:50:50.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:50:50.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:50:50.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:971 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (56) in=7b,truncate 3277332] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.3158799439999997, 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051333776999999997, 2024-04-20T02:50:50.911 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 971 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558678+0000", 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558686+0000", 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7330000000000003e-06 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558693+0000", 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.866e-06 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558698+0000", 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.49e-06 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594685+0000", 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035986753000000003 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594792+0000", 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107129 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594958+0000", 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000165429 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:50.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595721+0000", 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076360100000000004 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.596242+0000", 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052083800000000003 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609915+0000", 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013673431 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609958+0000", 2024-04-20T02:50:50.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2759000000000002e-05 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610013+0000", 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4421e-05 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.3158553680000002, 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.917 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:944 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'22, mlcod=30'17)", 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.559099+0000", 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.3154602220000005, 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0098123140000000008, 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.922 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559099+0000", 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559099+0000", 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559098+0000", 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559106+0000", 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.028e-06 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559116+0000", 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9310000000000003e-06 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.559122+0000", 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1299999999999998e-06 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.567846+0000", 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0087244330000000002 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.567871+0000", 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4865e-05 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568877+0000", 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010063870000000001 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568911+0000", 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3458999999999999e-05 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop_reply(client.4431.0:968 4.0 e32/22 ondisk, result = 0)", 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.3084150829999999, 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028443566999999999, 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566143+0000", 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566152+0000", 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1759999999999999e-06 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566163+0000", 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0852999999999999e-05 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566172+0000", 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2229999999999996e-06 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594564+0000", 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028392473000000001 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594574+0000", 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5899999999999997e-06 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594587+0000", 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3454e-05 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.929 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop_reply(client.4431.0:969 4.0 e32/22 ondisk, result = 0)", 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.578344+0000", 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.2962149199999997, 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016282119000000001, 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578344+0000", 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578344+0000", 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578343+0000", 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578354+0000", 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1241e-05 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578370+0000", 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6668999999999999e-05 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.578383+0000", 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2330999999999999e-05 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594611+0000", 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01622852 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594618+0000", 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0679999999999999e-06 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594626+0000", 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6729999999999992e-06 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.933 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:986 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'5, mlcod=29'3)", 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.2383543259999996, 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.040291053, 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:50.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636467+0000", 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026249800000000001 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636478+0000", 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1762e-05 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636493+0000", 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4826e-05 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636546+0000", 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2638e-05 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636570+0000", 2024-04-20T02:50:50.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4235000000000001e-05 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676457+0000", 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039887226999999997 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676496+0000", 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8281000000000002e-05 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:987 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'6, mlcod=29'3)", 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.651660+0000", 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.2228985799999998, 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026492596, 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651660+0000", 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651660+0000", 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651659+0000", 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651882+0000", 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000222222 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651896+0000", 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4541e-05 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651905+0000", 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1260000000000004e-06 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.657994+0000", 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0060883889999999996 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.658012+0000", 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7912000000000001e-05 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678124+0000", 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020112623999999999 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678153+0000", 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8632e-05 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.941 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:988 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'7, mlcod=29'3)", 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.667378+0000", 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.2071806059999997, 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011143905000000001, 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667378+0000", 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667378+0000", 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667377+0000", 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667399+0000", 2024-04-20T02:50:50.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.209e-05 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667410+0000", 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0906e-05 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667418+0000", 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7430000000000002e-06 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676540+0000", 2024-04-20T02:50:50.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0091222109999999999 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676565+0000", 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5196999999999999e-05 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678498+0000", 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00193249 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678522+0000", 2024-04-20T02:50:50.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4414999999999999e-05 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:989 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'8, mlcod=29'3)", 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.668591+0000", 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.2059674420000004, 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010451993, 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668591+0000", 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668591+0000", 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668591+0000", 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668597+0000", 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3589999999999996e-06 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668611+0000", 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3519e-05 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668617+0000", 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7839999999999998e-06 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677949+0000", 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093313839999999999 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.677973+0000", 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4127000000000001e-05 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679020+0000", 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001046644 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679043+0000", 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3771e-05 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1002 4.0 4:0158e114:::smithi10548823-21:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.1321205120000002, 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042474603999999999, 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:50.950 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1002 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742436+0000", 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742449+0000", 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3042e-05 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742469+0000", 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0007999999999999e-05 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742481+0000", 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1297999999999999e-05 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742528+0000", 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7697000000000001e-05 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742598+0000", 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9638e-05 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.784913+0000", 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042314928000000002 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.3.13:30 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.750440+0000", 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.1241183880000003, 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018760754000000001, 2024-04-20T02:50:50.954 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750440+0000", 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750440+0000", 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750439+0000", 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750701+0000", 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026198200000000001 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750716+0000", 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5128e-05 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750727+0000", 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1386000000000001e-05 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750782+0000", 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4985000000000003e-05 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.750944+0000", 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016174000000000001 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751215+0000", 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027062199999999998 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756304+0000", 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050896700000000001 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.757875+0000", 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001570788 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.769115+0000", 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011240333999999999 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.769165+0000", 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9784000000000003e-05 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.769201+0000", 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6004999999999998e-05 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.959 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.3.13:32 3.1 e32/21 3:aa868ed7:::8b42333603b5d66a962ab75ee4beb7799a2124b7175acbb12b87093577a68b6f:head v 32'86, mlcod=32'84)", 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.751861+0000", 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.1226980659999999, 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012698506, 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751861+0000", 2024-04-20T02:50:50.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751861+0000", 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751860+0000", 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751933+0000", 2024-04-20T02:50:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3002000000000003e-05 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751942+0000", 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9670000000000007e-06 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.751949+0000", 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4769999999999997e-06 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.752252+0000", 2024-04-20T02:50:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030264000000000002 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.752272+0000", 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0468000000000001e-05 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.764519+0000", 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012246309 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.764559+0000", 2024-04-20T02:50:50.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0543000000000001e-05 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.3.13:33 3.1 e32/21 3:886a183d:::3dba94e9eda505a2c38254a500ade87922ef0a7ff12c5928203f64ac9e8a3c99:head v 32'87, mlcod=32'84)", 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.756413+0000", 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.1181457979999996, 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0081679879999999993, 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:50.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756413+0000", 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756413+0000", 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756412+0000", 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756437+0000", 2024-04-20T02:50:50.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4346999999999999e-05 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756465+0000", 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8867999999999999e-05 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.756473+0000", 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.892e-06 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:50.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.760217+0000", 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037438910000000001 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.760240+0000", 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2980999999999999e-05 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.764562+0000", 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043219390000000003 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:50.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.764581+0000", 2024-04-20T02:50:50.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8859e-05 2024-04-20T02:50:50.968 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.968 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.968 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.968 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:50.968 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:50.968 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:50.969 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:50.969+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:50.970 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:50:50.972 INFO:tasks.rados.rados.0.smithi105.stdout:727: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-04-20T02:50:50.972 INFO:tasks.rados.rados.0.smithi105.stdout:727: writing smithi10548823-6 from 746244 to 1465388 tid 1 2024-04-20T02:50:50.972 INFO:tasks.rados.rados.0.smithi105.stdout:727: writing smithi10548823-6 from 2100461 to 2603823 tid 2 2024-04-20T02:50:50.972 INFO:tasks.rados.rados.0.smithi105.stdout:727: writing smithi10548823-6 from 3126680 to 3546244 tid 3 2024-04-20T02:50:50.972 INFO:tasks.rados.rados.0.smithi105.stdout:721: done (4 left) 2024-04-20T02:50:50.972 INFO:tasks.rados.rados.0.smithi105.stdout:722: done (3 left) 2024-04-20T02:50:50.972 INFO:tasks.rados.rados.0.smithi105.stdout:723: done (2 left) 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:726: done (1 left) 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:728: read oid 30 snap -1 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:728: expect (ObjNum 78 snap 1 seq_num 78) 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:729: read oid 8 snap -1 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:729: expect (ObjNum 86 snap 2 seq_num 86) 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:730: read oid 7 snap 2 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:730: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:731: tier_promote oid 40 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:732: write (excl) oid 49 current snap is 3 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:732: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:732: writing smithi10548823-49 from 763051 to 1557444 tid 1 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:732: writing smithi10548823-49 from 2158943 to 2805963 tid 2 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:732: writing smithi10548823-49 from 3437750 to 3563051 tid 3 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:733: write oid 20 current snap is 3 2024-04-20T02:50:50.973 INFO:tasks.rados.rados.0.smithi105.stdout:733: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-04-20T02:50:50.974 INFO:tasks.rados.rados.0.smithi105.stdout:733: writing smithi10548823-20 from 779858 to 1249500 tid 1 2024-04-20T02:50:50.974 INFO:tasks.rados.rados.0.smithi105.stdout:733: writing smithi10548823-20 from 1817425 to 2324456 tid 2 2024-04-20T02:50:50.975 INFO:tasks.rados.rados.0.smithi105.stdout:733: writing smithi10548823-20 from 2781526 to 3368154 tid 3 2024-04-20T02:50:50.975 INFO:tasks.rados.rados.0.smithi105.stdout:734: read oid 43 snap 2 2024-04-20T02:50:50.975 INFO:tasks.rados.rados.0.smithi105.stdout:734: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:50:50.975 INFO:tasks.rados.rados.0.smithi105.stdout:735: read oid 16 snap -1 2024-04-20T02:50:50.975 INFO:tasks.rados.rados.0.smithi105.stdout:735: expect deleted 2024-04-20T02:50:50.975 INFO:tasks.rados.rados.0.smithi105.stdout:736: read oid 37 snap -1 2024-04-20T02:50:50.975 INFO:tasks.rados.rados.0.smithi105.stdout:736: expect (ObjNum 91 snap 2 seq_num 91) 2024-04-20T02:50:50.975 INFO:tasks.rados.rados.0.smithi105.stdout:737: rollback oid 44 current snap is 3 2024-04-20T02:50:50.975 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 44 to 2 2024-04-20T02:50:50.975 INFO:tasks.rados.rados.0.smithi105.stdout:738: write oid 15 current snap is 3 2024-04-20T02:50:50.976 INFO:tasks.rados.rados.0.smithi105.stdout:728: expect (ObjNum 78 snap 1 seq_num 78) 2024-04-20T02:50:51.070 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:51.070+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:51.112 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:51.113 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:51.124 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:50:51.199 INFO:tasks.rados.rados.0.smithi105.stdout:729: expect (ObjNum 86 snap 2 seq_num 86) 2024-04-20T02:50:51.208 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:51.208+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:51.273 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:51.272+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:51.294 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:51.295 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:51.295 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:51.295 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:51.295 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:51.304 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:50:51.344 INFO:tasks.rados.rados.0.smithi105.stdout:727: finishing write tid 1 to smithi10548823-6 2024-04-20T02:50:51.344 INFO:tasks.rados.rados.0.smithi105.stdout:727: finishing write tid 2 to smithi10548823-6 2024-04-20T02:50:51.344 INFO:tasks.rados.rados.0.smithi105.stdout:738: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-04-20T02:50:51.351 INFO:tasks.rados.rados.0.smithi105.stdout:738: writing smithi10548823-15 from 796665 to 1341556 tid 1 2024-04-20T02:50:51.357 INFO:tasks.rados.rados.0.smithi105.stdout:738: writing smithi10548823-15 from 1759554 to 2410243 tid 2 2024-04-20T02:50:51.364 INFO:tasks.rados.rados.0.smithi105.stdout:738: writing smithi10548823-15 from 2976243 to 3596665 tid 3 2024-04-20T02:50:51.364 INFO:tasks.rados.rados.0.smithi105.stdout:728: done (11 left) 2024-04-20T02:50:51.364 INFO:tasks.rados.rados.0.smithi105.stdout:729: done (10 left) 2024-04-20T02:50:51.364 INFO:tasks.rados.rados.0.smithi105.stdout:739: read oid 42 snap 2 2024-04-20T02:50:51.364 INFO:tasks.rados.rados.0.smithi105.stdout:739: expect (ObjNum 41 snap 0 seq_num 41) 2024-04-20T02:50:51.364 INFO:tasks.rados.rados.0.smithi105.stdout:740: snap_create 2024-04-20T02:50:51.364 INFO:tasks.rados.rados.0.smithi105.stdout:730: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-20T02:50:51.374 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:50:51.388 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:51.388+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:51.443 INFO:tasks.rados.rados.0.smithi105.stdout:734: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:50:51.474 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:51.474+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:51.478 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:51.478 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:50:51.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:50:51.478 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:50:51.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.478 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:920 4.1 4:8efa5349:::smithi10548823-10:head [write 410104~465105] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.343140+0000", 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1342420989999997, 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039908454000000003, 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 920 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.343140+0000", 2024-04-20T02:50:51.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.343140+0000", 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.343137+0000", 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.343549+0000", 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000411939 2024-04-20T02:50:51.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.343560+0000", 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1486e-05 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.343566+0000", 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1109999999999996e-06 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.343620+0000", 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3860999999999998e-05 2024-04-20T02:50:51.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.343679+0000", 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8298999999999999e-05 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.343780+0000", 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101475 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.369299+0000", 2024-04-20T02:50:51.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025519304999999999 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.369348+0000", 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8508999999999999e-05 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.382963+0000", 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013615416 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.383013+0000", 2024-04-20T02:50:51.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0222999999999997e-05 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.383048+0000", 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4700000000000003e-05 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:921 4.1 4:8efa5349:::smithi10548823-10:head [write 1563527~583025] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.350674+0000", 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1267073920000001, 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035495006000000003, 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.484 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 921 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.350674+0000", 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.350674+0000", 2024-04-20T02:50:51.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.350673+0000", 2024-04-20T02:50:51.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:51.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.351253+0000", 2024-04-20T02:50:51.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00057907 2024-04-20T02:50:51.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.351265+0000", 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2403e-05 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.351288+0000", 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2674999999999999e-05 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.353992+0000", 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0027043229999999998 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.354058+0000", 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5875000000000002e-05 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.354171+0000", 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000113275 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.369401+0000", 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015230364 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.383056+0000", 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013654497999999999 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.386115+0000", 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030594440000000001 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.386145+0000", 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9703999999999999e-05 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.386169+0000", 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4303000000000001e-05 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(osd.0.15:86 3.3 3:fc8c1f37:::d6eead2e10707cc007893debcb7e3f1d2f0592877eb2179b53fb8b2234678183:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.0657005469999996, 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043953350000000002, 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:51.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:51.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:51.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:51.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411685+0000", 2024-04-20T02:50:51.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.763e-06 2024-04-20T02:50:51.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411696+0000", 2024-04-20T02:50:51.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0164e-05 2024-04-20T02:50:51.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411704+0000", 2024-04-20T02:50:51.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3599999999999996e-06 2024-04-20T02:50:51.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.412554+0000", 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00084986399999999998 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.412635+0000", 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.1284000000000003e-05 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.412785+0000", 2024-04-20T02:50:51.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015003799999999999 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.413631+0000", 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00084596599999999997 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.413863+0000", 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000231559 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.455552+0000", 2024-04-20T02:50:51.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041689853999999998 2024-04-20T02:50:51.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.455600+0000", 2024-04-20T02:50:51.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7352000000000003e-05 2024-04-20T02:50:51.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.455635+0000", 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4758999999999997e-05 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:941 4.c 4:392c1e2f:::smithi10548823-44:head [write 460525~690852,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.517365+0000", 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.960016553, 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043750506000000002, 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.499 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 941 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517365+0000", 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517365+0000", 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517361+0000", 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517909+0000", 2024-04-20T02:50:51.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00054796999999999995 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517931+0000", 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1642999999999999e-05 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517948+0000", 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6461999999999999e-05 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.518008+0000", 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0783999999999999e-05 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.518088+0000", 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9493000000000006e-05 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.518210+0000", 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012213400000000001 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.553216+0000", 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035006056000000001 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.559230+0000", 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060135939999999997 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.560955+0000", 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001725524 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.561052+0000", 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6465999999999994e-05 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.561116+0000", 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4140000000000006e-05 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:942 4.c 4:392c1e2f:::smithi10548823-44:head [write 1622620~730352] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.523837+0000", 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.9535443240000001, 2024-04-20T02:50:51.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044628446000000002, 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 942 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.523837+0000", 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.523837+0000", 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.523836+0000", 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.524505+0000", 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066824199999999999 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.524518+0000", 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3414000000000001e-05 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.524528+0000", 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0020999999999999e-05 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.534543+0000", 2024-04-20T02:50:51.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010014655000000001 2024-04-20T02:50:51.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.534645+0000", 2024-04-20T02:50:51.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010243600000000001 2024-04-20T02:50:51.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:50:51.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.534828+0000", 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182358 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.559305+0000", 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024477615000000001 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.560919+0000", 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001613511 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.568281+0000", 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0073622569999999997 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.568403+0000", 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012183700000000001 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.568466+0000", 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3138000000000004e-05 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:973 4.1 4:8efa5349:::smithi10548823-10:head [write 494139~557703,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.565018+0000", 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.9123641410000003, 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055306111999999998, 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.512 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 973 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565018+0000", 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565018+0000", 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565016+0000", 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565810+0000", 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00079402900000000002 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565823+0000", 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2568999999999999e-05 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565829+0000", 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5370000000000003e-06 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565866+0000", 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591000000000003e-05 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565927+0000", 2024-04-20T02:50:51.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0621000000000001e-05 2024-04-20T02:50:51.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:51.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.566035+0000", 2024-04-20T02:50:51.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010804 2024-04-20T02:50:51.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604559+0000", 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038524709999999997 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.612885+0000", 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0083253379999999998 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620219+0000", 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0073340899999999997 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620273+0000", 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4163999999999998e-05 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620324+0000", 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0934999999999999e-05 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:974 4.1 4:8efa5349:::smithi10548823-10:head [write 1455937~734021] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.572613+0000", 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.9047685049999998, 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047848988000000002, 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 974 2024-04-20T02:50:51.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.520 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.572613+0000", 2024-04-20T02:50:51.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.572613+0000", 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.572612+0000", 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.573688+0000", 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010766689999999999 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.573702+0000", 2024-04-20T02:50:51.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4003e-05 2024-04-20T02:50:51.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.573710+0000", 2024-04-20T02:50:51.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0849999999999997e-06 2024-04-20T02:50:51.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578846+0000", 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0051359739999999997 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578946+0000", 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9740000000000004e-05 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:51.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.579075+0000", 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000129102 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604563+0000", 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025487644 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.612927+0000", 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0083637120000000006 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620373+0000", 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074460020000000002 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620422+0000", 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9536999999999997e-05 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620462+0000", 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0082e-05 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:975 4.1 4:8efa5349:::smithi10548823-10:head [write 2948041~346098] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.578264+0000", 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.8991182960000002, 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042320522999999999, 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 975 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578264+0000", 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578264+0000", 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578262+0000", 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578837+0000", 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00057476800000000002 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578847+0000", 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9199999999999999e-06 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578855+0000", 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9149999999999996e-06 2024-04-20T02:50:51.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.595577+0000", 2024-04-20T02:50:51.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016722131000000001 2024-04-20T02:50:51.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.595661+0000", 2024-04-20T02:50:51.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4189000000000003e-05 2024-04-20T02:50:51.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:51.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.595799+0000", 2024-04-20T02:50:51.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013843300000000001 2024-04-20T02:50:51.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604568+0000", 2024-04-20T02:50:51.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087682909999999992 2024-04-20T02:50:51.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.613307+0000", 2024-04-20T02:50:51.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087388210000000008 2024-04-20T02:50:51.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620505+0000", 2024-04-20T02:50:51.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0071982119999999998 2024-04-20T02:50:51.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620546+0000", 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1325999999999997e-05 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620584+0000", 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7963000000000002e-05 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:976 4.1 4:8efa5349:::smithi10548823-10:head [setxattr _header (56) in=7b,truncate 3294139] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.533 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.578867+0000", 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.8985151169999996, 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042111858000000002, 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 976 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578867+0000", 2024-04-20T02:50:51.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578867+0000", 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578866+0000", 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578870+0000", 2024-04-20T02:50:51.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2080000000000002e-06 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578876+0000", 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2800000000000003e-06 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578881+0000", 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0300000000000001e-06 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604021+0000", 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025140466 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604120+0000", 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9167999999999996e-05 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604269+0000", 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014886500000000001 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.605069+0000", 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00080005199999999999 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.613490+0000", 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084207970000000007 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620868+0000", 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007377654 2024-04-20T02:50:51.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620922+0000", 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3934000000000002e-05 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620979+0000", 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7021999999999997e-05 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.539 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:977 4.1 4:8efa5349:::smithi10548823-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.578885+0000", 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.8984963639999997, 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042253767999999997, 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 977 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578885+0000", 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578885+0000", 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578885+0000", 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:51.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578887+0000", 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4310000000000001e-06 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578892+0000", 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9089999999999999e-06 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578897+0000", 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2880000000000003e-06 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604503+0000", 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025606278 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604551+0000", 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8322000000000003e-05 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.621039+0000", 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016487801 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.621085+0000", 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5748999999999999e-05 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.621139+0000", 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4415e-05 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:991 4.b e32/22 4:d4308fef:::smithi10548823-6:head v 32'29, mlcod=30'27)", 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.724859+0000", 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.7525225899999999, 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029751040999999999, 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.545 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.724859+0000", 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.724859+0000", 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.724859+0000", 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.725592+0000", 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00073373999999999998 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.725602+0000", 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8840000000000006e-06 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.725608+0000", 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7559999999999996e-06 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.725647+0000", 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8804000000000003e-05 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.725662+0000", 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5184000000000001e-05 2024-04-20T02:50:51.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.754577+0000", 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028915481999999999 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.754610+0000", 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2889000000000003e-05 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.549 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(osd.3.13:38 3.3 3:e41d6730:::cb7b5e1ce45104eea522c7c7be6c1af4d003bbae86351b50263a57d905ffb658:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.752689+0000", 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.7246932409999998, 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031765537000000003, 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.752689+0000", 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.752689+0000", 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.752688+0000", 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.752883+0000", 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019490499999999999 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.752894+0000", 2024-04-20T02:50:51.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1422e-05 2024-04-20T02:50:51.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.752902+0000", 2024-04-20T02:50:51.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2880000000000004e-06 2024-04-20T02:50:51.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.758085+0000", 2024-04-20T02:50:51.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0051837819999999996 2024-04-20T02:50:51.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.758227+0000", 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014176299999999999 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.758461+0000", 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023364999999999999 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.763297+0000", 2024-04-20T02:50:51.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048358740000000004 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.763489+0000", 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019218800000000001 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.784318+0000", 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020829008 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.784398+0000", 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9635000000000006e-05 2024-04-20T02:50:51.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.784454+0000", 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6505000000000002e-05 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1143 4.9 4:95e19a88:::smithi10548823-40:head [tier-promote] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.932855+0000", 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.54452638200000003, 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053775995, 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.556 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1143 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932855+0000", 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932855+0000", 2024-04-20T02:50:51.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932861+0000", 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5909999999999998e-06 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932878+0000", 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6081000000000001e-05 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932887+0000", 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7410000000000004e-06 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932954+0000", 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6551999999999996e-05 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.933079+0000", 2024-04-20T02:50:51.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012470600000000001 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986631+0000", 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053552884000000002 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.53440479699999999, 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:50:51.560 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:50:51.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:50:51.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:50:51.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:51.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:50:51.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:50:51.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:50:51.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:50:51.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:50:51.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:50:51.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:51.567 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:50:51.568 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.526926479, 2024-04-20T02:50:51.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:50:51.568 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.568 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.568 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.568 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.568 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.568 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:50:51.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:51.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:50:51.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:50:51.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:50:51.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:50:51.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:50:51.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:50:51.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:50:51.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1146 4.d 4:b376c20e:::smithi10548823-49:head [write 3437750~125301] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:50:51.575 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.52235861800000005, 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070707487999999999, 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1146 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955022+0000", 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955179+0000", 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015657 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955188+0000", 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.645e-06 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955195+0000", 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2910000000000004e-06 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978329+0000", 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02313401 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978450+0000", 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121438 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978652+0000", 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020163700000000001 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984380+0000", 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057282510000000002 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991325+0000", 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069446830000000001 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025680+0000", 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034355284 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025709+0000", 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8781000000000001e-05 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025731+0000", 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1963e-05 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1147 4.d 4:b376c20e:::smithi10548823-49:head [setxattr _header (56) in=7b,truncate 3563051] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.52218028100000002, 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070713812000000001, 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.581 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.582 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.582 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.582 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1147 2024-04-20T02:50:51.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.582 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:50:51.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955201+0000", 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955204+0000", 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.05e-06 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955207+0000", 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1949999999999999e-06 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955210+0000", 2024-04-20T02:50:51.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:50:51.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983351+0000", 2024-04-20T02:50:51.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028140978000000001 2024-04-20T02:50:51.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983488+0000", 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013647 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983668+0000", 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000180335 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984487+0000", 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081847099999999998 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991393+0000", 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069068569999999998 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025852+0000", 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034458946999999997 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025883+0000", 2024-04-20T02:50:51.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0261e-05 2024-04-20T02:50:51.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025915+0000", 2024-04-20T02:50:51.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2654999999999998e-05 2024-04-20T02:50:51.588 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.588 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.588 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.52216857999999999, 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:50:51.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:50:51.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:51.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:50:51.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1149 4.b e35/22 4:de892c9b:::smithi10548823-20:head v 35'45, mlcod=35'41)", 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.972235+0000", 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.50514672100000002, 2024-04-20T02:50:51.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039533568999999998, 2024-04-20T02:50:51.595 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.595 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.595 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.972235+0000", 2024-04-20T02:50:51.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.972235+0000", 2024-04-20T02:50:51.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.972235+0000", 2024-04-20T02:50:51.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:51.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.972658+0000", 2024-04-20T02:50:51.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000423304 2024-04-20T02:50:51.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.972667+0000", 2024-04-20T02:50:51.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3500000000000003e-06 2024-04-20T02:50:51.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.972673+0000", 2024-04-20T02:50:51.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4410000000000001e-06 2024-04-20T02:50:51.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.972716+0000", 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2954000000000002e-05 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.972740+0000", 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4771000000000001e-05 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.011725+0000", 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038984431 2024-04-20T02:50:51.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.011769+0000", 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3887000000000002e-05 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1150 4.b e35/22 4:de892c9b:::smithi10548823-20:head v 35'46, mlcod=35'41)", 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.980947+0000", 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.49643436400000002, 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030857921, 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:51.600 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.980947+0000", 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.980947+0000", 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.980947+0000", 2024-04-20T02:50:51.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:51.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:51.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983223+0000", 2024-04-20T02:50:51.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022755890000000002 2024-04-20T02:50:51.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983238+0000", 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5868000000000001e-05 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983249+0000", 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0377000000000001e-05 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986539+0000", 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003289798 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986558+0000", 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9089e-05 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.011784+0000", 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025226624999999999 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.011805+0000", 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0999e-05 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.605 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:51.605 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:51.605 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:51.606 INFO:tasks.rados.rados.0.smithi105.stdout:737: finishing rollback tid 0 to smithi10548823-44 2024-04-20T02:50:51.606 INFO:tasks.rados.rados.0.smithi105.stdout:731: got (0) Success 2024-04-20T02:50:51.606 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 21 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:51.606 INFO:tasks.rados.rados.0.smithi105.stdout:737: finishing rollback tid 1 to smithi10548823-44 2024-04-20T02:50:51.607 INFO:tasks.rados.rados.0.smithi105.stdout:737: finishing rollback tid 2 to smithi10548823-44 2024-04-20T02:50:51.607 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 30 (ObjNum 75 snap 1 seq_num 75) dirty exists 2024-04-20T02:50:51.607 INFO:tasks.rados.rados.0.smithi105.stdout:736: expect (ObjNum 91 snap 2 seq_num 91) 2024-04-20T02:50:51.607 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:51.606+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:51.607 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:50:51.707 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:51.707+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:51.739 INFO:tasks.rados.rados.0.smithi105.stdout:727: finishing write tid 3 to smithi10548823-6 2024-04-20T02:50:51.739 INFO:tasks.rados.rados.0.smithi105.stdout:727: finishing write tid 4 to smithi10548823-6 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:727: finishing write tid 5 to smithi10548823-6 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:727: oid 6 updating version 0 to 42 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:727: oid 6 version 42 is already newer than 41 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:727: oid 6 updating version 42 to 43 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:727: oid 6 updating version 43 to 44 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 44 (ObjNum 92 snap 3 seq_num 92) dirty exists 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:727: left oid 6 (ObjNum 92 snap 3 seq_num 92) 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:733: finishing write tid 1 to smithi10548823-20 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:733: finishing write tid 2 to smithi10548823-20 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:733: finishing write tid 3 to smithi10548823-20 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:733: finishing write tid 4 to smithi10548823-20 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:733: finishing write tid 5 to smithi10548823-20 2024-04-20T02:50:51.740 INFO:tasks.rados.rados.0.smithi105.stdout:733: oid 20 updating version 0 to 45 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:733: oid 20 updating version 45 to 46 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:733: oid 20 updating version 46 to 48 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:733: oid 20 version 48 is already newer than 47 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 48 (ObjNum 94 snap 3 seq_num 94) dirty exists 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:733: left oid 20 (ObjNum 94 snap 3 seq_num 94) 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:732: finishing write tid 1 to smithi10548823-49 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:732: finishing write tid 2 to smithi10548823-49 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:732: finishing write tid 3 to smithi10548823-49 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:732: finishing write tid 4 to smithi10548823-49 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:732: finishing write tid 5 to smithi10548823-49 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:732: oid 49 updating version 0 to 29 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:732: oid 49 updating version 29 to 32 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:732: oid 49 version 32 is already newer than 31 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:732: oid 49 version 32 is already newer than 30 2024-04-20T02:50:51.741 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 32 (ObjNum 93 snap 3 seq_num 93) dirty exists 2024-04-20T02:50:51.742 INFO:tasks.rados.rados.0.smithi105.stdout:732: left oid 49 (ObjNum 93 snap 3 seq_num 93) 2024-04-20T02:50:51.742 INFO:tasks.rados.rados.0.smithi105.stdout:739: expect (ObjNum 41 snap 0 seq_num 41) 2024-04-20T02:50:51.777 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:51.777 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:51.777 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:51.777 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:51.787 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:50:51.826 INFO:tasks.rados.rados.0.smithi105.stdout:738: finishing write tid 1 to smithi10548823-15 2024-04-20T02:50:51.826 INFO:tasks.rados.rados.0.smithi105.stdout:738: finishing write tid 2 to smithi10548823-15 2024-04-20T02:50:51.826 INFO:tasks.rados.rados.0.smithi105.stdout:738: finishing write tid 3 to smithi10548823-15 2024-04-20T02:50:51.826 INFO:tasks.rados.rados.0.smithi105.stdout:738: finishing write tid 4 to smithi10548823-15 2024-04-20T02:50:51.826 INFO:tasks.rados.rados.0.smithi105.stdout:738: finishing write tid 5 to smithi10548823-15 2024-04-20T02:50:51.826 INFO:tasks.rados.rados.0.smithi105.stdout:738: oid 15 updating version 0 to 54 2024-04-20T02:50:51.826 INFO:tasks.rados.rados.0.smithi105.stdout:738: oid 15 version 54 is already newer than 53 2024-04-20T02:50:51.826 INFO:tasks.rados.rados.0.smithi105.stdout:738: oid 15 version 54 is already newer than 51 2024-04-20T02:50:51.826 INFO:tasks.rados.rados.0.smithi105.stdout:738: oid 15 version 54 is already newer than 52 2024-04-20T02:50:51.827 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 54 (ObjNum 95 snap 3 seq_num 95) dirty exists 2024-04-20T02:50:51.827 INFO:tasks.rados.rados.0.smithi105.stdout:738: left oid 15 (ObjNum 95 snap 3 seq_num 95) 2024-04-20T02:50:51.828 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:50:51.829 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:51.828+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:51.909 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:51.909+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:51.956 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:51.956 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:51.956 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:51.956 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:51.956 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:51.967 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:50:51.973 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:50:51.991 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:50:52.010 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:50:52.010 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:52.010+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:52.029 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:50:52.049 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:50:52.068 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:50:52.087 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:50:52.087 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:50:52.087 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:50:52.088 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:50:52.088 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:50:52.088 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:50:52.088 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-20T02:50:52.088 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:50:52.088 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:50:52.090 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:50:52.090 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:50:52.097 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:52.097 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:52.110 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:50:52.151 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:52.150+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:52.212 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:52.213+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:52.283 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:52.283 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:52.293 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:50:52.313 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:52.314+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:52.349 INFO:tasks.rados.rados.0.smithi105.stdout:727: done (11 left) 2024-04-20T02:50:52.350 INFO:tasks.rados.rados.0.smithi105.stdout:730: done (10 left) 2024-04-20T02:50:52.350 INFO:tasks.rados.rados.0.smithi105.stdout:731: done (9 left) 2024-04-20T02:50:52.350 INFO:tasks.rados.rados.0.smithi105.stdout:732: done (8 left) 2024-04-20T02:50:52.350 INFO:tasks.rados.rados.0.smithi105.stdout:733: done (7 left) 2024-04-20T02:50:52.350 INFO:tasks.rados.rados.0.smithi105.stdout:734: done (6 left) 2024-04-20T02:50:52.350 INFO:tasks.rados.rados.0.smithi105.stdout:735: done (5 left) 2024-04-20T02:50:52.350 INFO:tasks.rados.rados.0.smithi105.stdout:736: done (4 left) 2024-04-20T02:50:52.350 INFO:tasks.rados.rados.0.smithi105.stdout:737: done (3 left) 2024-04-20T02:50:52.351 INFO:tasks.rados.rados.0.smithi105.stdout:738: done (2 left) 2024-04-20T02:50:52.351 INFO:tasks.rados.rados.0.smithi105.stdout:739: done (1 left) 2024-04-20T02:50:52.351 INFO:tasks.rados.rados.0.smithi105.stdout:740: done (0 left) 2024-04-20T02:50:52.351 INFO:tasks.rados.rados.0.smithi105.stdout:741: tier_promote oid 40 2024-04-20T02:50:52.351 INFO:tasks.rados.rados.0.smithi105.stdout:742: tier_promote oid 27 2024-04-20T02:50:52.351 INFO:tasks.rados.rados.0.smithi105.stdout:743: write (excl) oid 4 current snap is 4 2024-04-20T02:50:52.351 INFO:tasks.rados.rados.0.smithi105.stdout:743: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-04-20T02:50:52.352 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:50:52.355 INFO:tasks.rados.rados.0.smithi105.stdout:743: writing smithi10548823-4 from 413472 to 1033612 tid 1 2024-04-20T02:50:52.363 INFO:tasks.rados.rados.0.smithi105.stdout:743: writing smithi10548823-4 from 1818036 to 2328736 tid 2 2024-04-20T02:50:52.370 INFO:tasks.rados.rados.0.smithi105.stdout:743: writing smithi10548823-4 from 3120019 to 3613472 tid 3 2024-04-20T02:50:52.370 INFO:tasks.rados.rados.0.smithi105.stdout:744: write oid 13 current snap is 4 2024-04-20T02:50:52.370 INFO:tasks.rados.rados.0.smithi105.stdout:744: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-04-20T02:50:52.376 INFO:tasks.rados.rados.0.smithi105.stdout:744: writing smithi10548823-13 from 430279 to 1125668 tid 1 2024-04-20T02:50:52.385 INFO:tasks.rados.rados.0.smithi105.stdout:744: writing smithi10548823-13 from 1876518 to 2530876 tid 2 2024-04-20T02:50:52.391 INFO:tasks.rados.rados.0.smithi105.stdout:744: writing smithi10548823-13 from 3031089 to 3567886 tid 3 2024-04-20T02:50:52.391 INFO:tasks.rados.rados.0.smithi105.stdout:745: delete oid 15 current snap is 4 2024-04-20T02:50:52.391 INFO:tasks.rados.rados.0.smithi105.stdout:742: got (0) Success 2024-04-20T02:50:52.391 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 37 (ObjNum 84 snap 2 seq_num 84) dirty exists 2024-04-20T02:50:52.409 INFO:tasks.rados.rados.0.smithi105.stdout:743: finishing write tid 1 to smithi10548823-4 2024-04-20T02:50:52.410 INFO:tasks.rados.rados.0.smithi105.stdout:743: finishing write tid 2 to smithi10548823-4 2024-04-20T02:50:52.411 INFO:tasks.rados.rados.0.smithi105.stdout:743: finishing write tid 3 to smithi10548823-4 2024-04-20T02:50:52.411 INFO:tasks.rados.rados.0.smithi105.stdout:743: finishing write tid 4 to smithi10548823-4 2024-04-20T02:50:52.411 INFO:tasks.rados.rados.0.smithi105.stdout:743: finishing write tid 5 to smithi10548823-4 2024-04-20T02:50:52.412 INFO:tasks.rados.rados.0.smithi105.stdout:743: oid 4 updating version 0 to 69 2024-04-20T02:50:52.412 INFO:tasks.rados.rados.0.smithi105.stdout:743: oid 4 version 69 is already newer than 67 2024-04-20T02:50:52.412 INFO:tasks.rados.rados.0.smithi105.stdout:743: oid 4 updating version 69 to 70 2024-04-20T02:50:52.412 INFO:tasks.rados.rados.0.smithi105.stdout:743: oid 4 version 70 is already newer than 68 2024-04-20T02:50:52.412 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 70 (ObjNum 96 snap 4 seq_num 96) dirty exists 2024-04-20T02:50:52.412 INFO:tasks.rados.rados.0.smithi105.stdout:743: left oid 4 (ObjNum 96 snap 4 seq_num 96) 2024-04-20T02:50:52.415 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:52.415+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:52.420 INFO:tasks.rados.rados.0.smithi105.stdout:741: got (0) Success 2024-04-20T02:50:52.421 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 21 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-20T02:50:52.421 INFO:tasks.rados.rados.0.smithi105.stdout:744: finishing write tid 1 to smithi10548823-13 2024-04-20T02:50:52.433 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:52.433 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:52.433 INFO:tasks.rados.rados.0.smithi105.stdout:744: finishing write tid 2 to smithi10548823-13 2024-04-20T02:50:52.438 INFO:tasks.rados.rados.0.smithi105.stdout:744: finishing write tid 3 to smithi10548823-13 2024-04-20T02:50:52.438 INFO:tasks.rados.rados.0.smithi105.stdout:744: finishing write tid 4 to smithi10548823-13 2024-04-20T02:50:52.438 INFO:tasks.rados.rados.0.smithi105.stdout:744: finishing write tid 5 to smithi10548823-13 2024-04-20T02:50:52.438 INFO:tasks.rados.rados.0.smithi105.stdout:744: oid 13 updating version 0 to 58 2024-04-20T02:50:52.438 INFO:tasks.rados.rados.0.smithi105.stdout:744: oid 13 updating version 58 to 60 2024-04-20T02:50:52.438 INFO:tasks.rados.rados.0.smithi105.stdout:744: oid 13 updating version 60 to 61 2024-04-20T02:50:52.438 INFO:tasks.rados.rados.0.smithi105.stdout:744: oid 13 version 61 is already newer than 59 2024-04-20T02:50:52.438 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 61 (ObjNum 97 snap 4 seq_num 97) dirty exists 2024-04-20T02:50:52.438 INFO:tasks.rados.rados.0.smithi105.stdout:744: left oid 13 (ObjNum 97 snap 4 seq_num 97) 2024-04-20T02:50:52.438 INFO:tasks.rados.rados.0.smithi105.stdout:741: done (4 left) 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:742: done (3 left) 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:743: done (2 left) 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:744: done (1 left) 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:745: done (0 left) 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:746: read oid 27 snap 2 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:746: expect (ObjNum 84 snap 2 seq_num 84) 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:747: tier_flush oid 48 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:747: tier_flush oid 48 snap -1 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:748: read oid 4 snap -1 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:748: expect (ObjNum 96 snap 4 seq_num 96) 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:749: read oid 12 snap 2 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:749: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:750: read oid 6 snap -1 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:750: expect (ObjNum 92 snap 3 seq_num 92) 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:751: read oid 22 snap -1 2024-04-20T02:50:52.439 INFO:tasks.rados.rados.0.smithi105.stdout:751: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-20T02:50:52.440 INFO:tasks.rados.rados.0.smithi105.stdout:752: read oid 5 snap -1 2024-04-20T02:50:52.440 INFO:tasks.rados.rados.0.smithi105.stdout:752: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-20T02:50:52.440 INFO:tasks.rados.rados.0.smithi105.stdout:753: snap_remove snap 3 2024-04-20T02:50:52.445 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:50:52.516 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:52.515+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:52.583 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:52.583 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:52.594 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:50:52.639 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:52.639+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:52.718 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:52.718+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:52.722 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:52.723 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:52.733 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:50:52.818 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:52.818+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:52.882 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:52.881+0000 7fcba0c807c0 -1 osd.5 25 log_to_monitors true 2024-04-20T02:50:52.902 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:52.902 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:52.912 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:50:52.953 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:52.953+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:53.020 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:53.020+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:53.080 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:53.080 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:53.090 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:50:53.132 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:53.132+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:53.222 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:53.222+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:53.259 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:53.259 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:53.270 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:50:53.323 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:53.323+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:53.349 INFO:tasks.rados.rados.0.smithi105.stdout:753: done (7 left) 2024-04-20T02:50:53.349 INFO:tasks.rados.rados.0.smithi105.stdout:754: snap_create 2024-04-20T02:50:53.349 INFO:tasks.rados.rados.0.smithi105.stdout:749: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-20T02:50:53.356 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:53.356+0000 7fcb8cc12700 -1 osd.5 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:50:53.423 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:53.422+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:53.443 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:53.443 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:53.453 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:50:53.492 INFO:tasks.rados.rados.0.smithi105.stdout:746: expect (ObjNum 84 snap 2 seq_num 84) 2024-04-20T02:50:53.524 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:53.524+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:53.622 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:53.623 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:53.625 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:53.625+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:53.633 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:50:53.643 INFO:tasks.rados.rados.0.smithi105.stdout:752: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-20T02:50:53.712 INFO:tasks.rados.rados.0.smithi105.stdout:748: expect (ObjNum 96 snap 4 seq_num 96) 2024-04-20T02:50:53.726 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:53.726+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:53.762 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:53.762 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:53.773 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:50:53.857 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:53.857+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:53.928 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:53.927+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:53.935 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:50:53.943 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:53.943 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:53.946 INFO:tasks.rados.rados.0.smithi105.stdout:750: expect (ObjNum 92 snap 3 seq_num 92) 2024-04-20T02:50:53.953 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:50:54.038 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:54.038+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:54.078 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:54.078 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:54.078 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:54.078 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:54.088 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-04-20T02:50:54.092 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:50:54.124 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:54.124 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:54.130 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:54.130+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:50:54.135 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:50:54.191 INFO:tasks.rados.rados.0.smithi105.stdout:751: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-20T02:50:54.231 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:54.231+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:54.240 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:50:54.261 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:50:54.275 INFO:tasks.rados.rados.0.smithi105.stdout:747: got (0) Success 2024-04-20T02:50:54.275 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 0 (ObjNum 82 snap 1 seq_num 82) dirty exists 2024-04-20T02:50:54.280 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:50:54.300 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:50:54.308 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:54.308 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:54.308 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:50:54.308 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:54.319 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:50:54.320 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:50:54.332 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:54.333+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:54.340 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:50:54.357 INFO:tasks.rados.rados.0.smithi105.stdout:746: done (7 left) 2024-04-20T02:50:54.357 INFO:tasks.rados.rados.0.smithi105.stdout:747: done (6 left) 2024-04-20T02:50:54.358 INFO:tasks.rados.rados.0.smithi105.stdout:748: done (5 left) 2024-04-20T02:50:54.358 INFO:tasks.rados.rados.0.smithi105.stdout:749: done (4 left) 2024-04-20T02:50:54.358 INFO:tasks.rados.rados.0.smithi105.stdout:750: done (3 left) 2024-04-20T02:50:54.358 INFO:tasks.rados.rados.0.smithi105.stdout:751: done (2 left) 2024-04-20T02:50:54.358 INFO:tasks.rados.rados.0.smithi105.stdout:752: done (1 left) 2024-04-20T02:50:54.358 INFO:tasks.rados.rados.0.smithi105.stdout:754: done (0 left) 2024-04-20T02:50:54.358 INFO:tasks.rados.rados.0.smithi105.stdout:755: write (excl) oid 14 current snap is 5 2024-04-20T02:50:54.358 INFO:tasks.rados.rados.0.smithi105.stdout:755: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-04-20T02:50:54.363 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:50:54.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:50:54.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:50:54.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:50:54.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:50:54.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:50:54.363 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-20T02:50:54.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:50:54.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:50:54.364 INFO:tasks.rados.rados.0.smithi105.stdout:755: writing smithi10548823-14 from 463893 to 1026133 tid 1 2024-04-20T02:50:54.370 INFO:tasks.rados.rados.0.smithi105.stdout:755: writing smithi10548823-14 from 1593482 to 2251509 tid 2 2024-04-20T02:50:54.372 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:50:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:50:54.377 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:50:54.377 INFO:tasks.rados.rados.0.smithi105.stdout:755: writing smithi10548823-14 from 2685935 to 3245276 tid 3 2024-04-20T02:50:54.377 INFO:tasks.rados.rados.0.smithi105.stdout:756: rollback oid 9 current snap is 5 2024-04-20T02:50:54.377 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 9 to 4 2024-04-20T02:50:54.377 INFO:tasks.rados.rados.0.smithi105.stdout:757: rollback oid 28 current snap is 5 2024-04-20T02:50:54.378 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 28 to 2 2024-04-20T02:50:54.378 INFO:tasks.rados.rados.0.smithi105.stdout:758: tier_promote oid 36 2024-04-20T02:50:54.378 INFO:tasks.rados.rados.0.smithi105.stdout:759: write (excl) oid 40 current snap is 5 2024-04-20T02:50:54.378 INFO:tasks.rados.rados.0.smithi105.stdout:759: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-04-20T02:50:54.383 INFO:tasks.rados.rados.0.smithi105.stdout:759: writing smithi10548823-40 from 480700 to 1118189 tid 1 2024-04-20T02:50:54.389 INFO:tasks.rados.rados.0.smithi105.stdout:759: writing smithi10548823-40 from 1651964 to 2053649 tid 2 2024-04-20T02:50:54.398 INFO:tasks.rados.rados.0.smithi105.stdout:759: writing smithi10548823-40 from 2713358 to 3483971 tid 3 2024-04-20T02:50:54.398 INFO:tasks.rados.rados.0.smithi105.stdout:760: tier_evict oid 30 2024-04-20T02:50:54.398 INFO:tasks.rados.rados.0.smithi105.stdout:760: tier_evict oid 30 snap -1 2024-04-20T02:50:54.398 INFO:tasks.rados.rados.0.smithi105.stdout:756: finishing rollback tid 0 to smithi10548823-9 2024-04-20T02:50:54.398 INFO:tasks.rados.rados.0.smithi105.stdout:761: read oid 44 snap 4 2024-04-20T02:50:54.398 INFO:tasks.rados.rados.0.smithi105.stdout:761: expect (ObjNum 75 snap 1 seq_num 75) 2024-04-20T02:50:54.399 INFO:tasks.rados.rados.0.smithi105.stdout:762: write (excl) oid 32 current snap is 5 2024-04-20T02:50:54.399 INFO:tasks.rados.rados.0.smithi105.stdout:762: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-04-20T02:50:54.414 INFO:tasks.rados.rados.0.smithi105.stdout:762: writing smithi10548823-32 from 497507 to 1210245 tid 1 2024-04-20T02:50:54.420 INFO:tasks.rados.rados.0.smithi105.stdout:762: writing smithi10548823-32 from 1710446 to 2372142 tid 2 2024-04-20T02:50:54.423 INFO:tasks.rados.rados.0.smithi105.stdout:762: writing smithi10548823-32 from 3140781 to 3697507 tid 3 2024-04-20T02:50:54.423 INFO:tasks.rados.rados.0.smithi105.stdout:763: delete oid 50 current snap is 5 2024-04-20T02:50:54.423 INFO:tasks.rados.rados.0.smithi105.stdout:756: finishing rollback tid 1 to smithi10548823-9 2024-04-20T02:50:54.423 INFO:tasks.rados.rados.0.smithi105.stdout:756: finishing rollback tid 2 to smithi10548823-9 2024-04-20T02:50:54.423 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 76 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-04-20T02:50:54.424 INFO:tasks.rados.rados.0.smithi105.stdout:760: got (22) Invalid argument 2024-04-20T02:50:54.424 INFO:tasks.rados.rados.0.smithi105.stdout:757: finishing rollback tid 0 to smithi10548823-28 2024-04-20T02:50:54.424 INFO:tasks.rados.rados.0.smithi105.stdout:761: expect (ObjNum 75 snap 1 seq_num 75) 2024-04-20T02:50:54.434 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:54.433+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:54.498 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:54.498 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:54.498 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:50:54.498 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:50:54.498 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:54.508 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:50:54.534 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:54.533+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:54.629 INFO:tasks.rados.rados.0.smithi105.stdout:755: finishing write tid 1 to smithi10548823-14 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:755: finishing write tid 2 to smithi10548823-14 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:755: finishing write tid 3 to smithi10548823-14 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:755: finishing write tid 4 to smithi10548823-14 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:756: done (8 left) 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:760: done (7 left) 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:761: done (6 left) 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:763: done (5 left) 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:764: tier_evict oid 21 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:764: tier_evict oid 21 snap -1 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:755: finishing write tid 5 to smithi10548823-14 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:755: oid 14 updating version 0 to 39 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:755: oid 14 version 39 is already newer than 38 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:755: oid 14 updating version 39 to 41 2024-04-20T02:50:54.630 INFO:tasks.rados.rados.0.smithi105.stdout:755: oid 14 version 41 is already newer than 40 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 41 (ObjNum 99 snap 5 seq_num 99) dirty exists 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:755: left oid 14 (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:762: finishing write tid 1 to smithi10548823-32 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:762: finishing write tid 2 to smithi10548823-32 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:762: finishing write tid 3 to smithi10548823-32 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:762: finishing write tid 4 to smithi10548823-32 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:762: finishing write tid 5 to smithi10548823-32 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:762: oid 32 updating version 0 to 11 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:762: oid 32 updating version 11 to 12 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:762: oid 32 updating version 12 to 14 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:762: oid 32 version 14 is already newer than 13 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 14 (ObjNum 101 snap 5 seq_num 101) dirty exists 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:762: left oid 32 (ObjNum 101 snap 5 seq_num 101) 2024-04-20T02:50:54.631 INFO:tasks.rados.rados.0.smithi105.stdout:757: finishing rollback tid 1 to smithi10548823-28 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:757: finishing rollback tid 2 to smithi10548823-28 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 53 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:759: finishing write tid 1 to smithi10548823-40 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:759: finishing write tid 2 to smithi10548823-40 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:758: got (0) Success 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 18 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:759: finishing write tid 3 to smithi10548823-40 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:759: finishing write tid 4 to smithi10548823-40 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:755: done (5 left) 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:757: done (4 left) 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:758: done (3 left) 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:762: done (2 left) 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:765: read oid 43 snap -1 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:765: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:766: write oid 27 current snap is 5 2024-04-20T02:50:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:759: finishing write tid 5 to smithi10548823-40 2024-04-20T02:50:54.633 INFO:tasks.rados.rados.0.smithi105.stdout:759: oid 40 updating version 0 to 74 2024-04-20T02:50:54.633 INFO:tasks.rados.rados.0.smithi105.stdout:759: oid 40 version 74 is already newer than 69 2024-04-20T02:50:54.633 INFO:tasks.rados.rados.0.smithi105.stdout:759: oid 40 version 74 is already newer than 70 2024-04-20T02:50:54.633 INFO:tasks.rados.rados.0.smithi105.stdout:759: oid 40 version 74 is already newer than 73 2024-04-20T02:50:54.633 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 74 (ObjNum 100 snap 5 seq_num 100) dirty exists 2024-04-20T02:50:54.633 INFO:tasks.rados.rados.0.smithi105.stdout:759: left oid 40 (ObjNum 100 snap 5 seq_num 100) 2024-04-20T02:50:54.633 INFO:tasks.rados.rados.0.smithi105.stdout:766: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-04-20T02:50:54.634 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:54.634+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:54.635 INFO:tasks.rados.rados.0.smithi105.stdout:766: writing smithi10548823-27 from 531121 to 994357 tid 1 2024-04-20T02:50:54.644 INFO:tasks.rados.rados.0.smithi105.stdout:766: writing smithi10548823-27 from 1711057 to 2376422 tid 2 2024-04-20T02:50:54.653 INFO:tasks.rados.rados.0.smithi105.stdout:766: writing smithi10548823-27 from 3079274 to 3731121 tid 3 2024-04-20T02:50:54.653 INFO:tasks.rados.rados.0.smithi105.stdout:759: done (3 left) 2024-04-20T02:50:54.653 INFO:tasks.rados.rados.0.smithi105.stdout:767: read oid 16 snap -1 2024-04-20T02:50:54.653 INFO:tasks.rados.rados.0.smithi105.stdout:767: expect deleted 2024-04-20T02:50:54.653 INFO:tasks.rados.rados.0.smithi105.stdout:768: rollback oid 6 current snap is 5 2024-04-20T02:50:54.653 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 6 to 4 2024-04-20T02:50:54.653 INFO:tasks.rados.rados.0.smithi105.stdout:769: read oid 24 snap -1 2024-04-20T02:50:54.653 INFO:tasks.rados.rados.0.smithi105.stdout:769: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-20T02:50:54.653 INFO:tasks.rados.rados.0.smithi105.stdout:770: tier_evict oid 20 2024-04-20T02:50:54.653 INFO:tasks.rados.rados.0.smithi105.stdout:770: tier_evict oid 20 snap -1 2024-04-20T02:50:54.654 INFO:tasks.rados.rados.0.smithi105.stdout:771: delete oid 28 current snap is 5 2024-04-20T02:50:54.654 INFO:tasks.rados.rados.0.smithi105.stdout:764: got (22) Invalid argument 2024-04-20T02:50:54.654 INFO:tasks.rados.rados.0.smithi105.stdout:765: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:941 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'19, mlcod=30'17)", 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.1648777890000002, 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042482077, 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.684 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518290+0000", 2024-04-20T02:50:54.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518988+0000", 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069878099999999999 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519003+0000", 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4756999999999999e-05 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519013+0000", 2024-04-20T02:50:54.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4250000000000004e-06 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519071+0000", 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8628000000000002e-05 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519106+0000", 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4817000000000002e-05 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560724+0000", 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041618334 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560773+0000", 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8937999999999998e-05 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:942 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'20, mlcod=30'17)", 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.1460606799999997, 2024-04-20T02:50:54.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030979784999999999, 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537107+0000", 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538354+0000", 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001247232 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538381+0000", 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7008999999999998e-05 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538395+0000", 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4093999999999999e-05 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538476+0000", 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1045999999999996e-05 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538505+0000", 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8969999999999999e-05 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568053+0000", 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029547370999999999 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568088+0000", 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5540000000000002e-05 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:54.692 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.1394294540000001, 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:50:54.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:50:54.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:50:54.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:54.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:50:54.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:50:54.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:54.699 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.1332083799999992, 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:50:54.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:50:54.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:50:54.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.1248081760000002, 2024-04-20T02:50:54.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:54.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:971 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (56) in=7b,truncate 3277332] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.1244901069999997, 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051333776999999997, 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.711 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 971 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558678+0000", 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558686+0000", 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7330000000000003e-06 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558693+0000", 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.866e-06 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558698+0000", 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.49e-06 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594685+0000", 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035986753000000003 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594792+0000", 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107129 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594958+0000", 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000165429 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595721+0000", 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076360100000000004 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.596242+0000", 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052083800000000003 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609915+0000", 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013673431 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609958+0000", 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2759000000000002e-05 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610013+0000", 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4421e-05 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.1244655310000002, 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.717 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:50:54.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.721 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop_reply(client.4431.0:968 4.0 e32/22 ondisk, result = 0)", 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.1170252460000007, 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028443566999999999, 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566143+0000", 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566152+0000", 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1759999999999999e-06 2024-04-20T02:50:54.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566163+0000", 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0852999999999999e-05 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566172+0000", 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2229999999999996e-06 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594564+0000", 2024-04-20T02:50:54.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028392473000000001 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594574+0000", 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5899999999999997e-06 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594587+0000", 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3454e-05 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:986 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'5, mlcod=29'3)", 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.0469644890000005, 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.040291053, 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636467+0000", 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026249800000000001 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636478+0000", 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1762e-05 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636493+0000", 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4826e-05 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636546+0000", 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2638e-05 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636570+0000", 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4235000000000001e-05 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676457+0000", 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039887226999999997 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676496+0000", 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8281000000000002e-05 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.729 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:987 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'6, mlcod=29'3)", 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.651660+0000", 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.0315087429999998, 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026492596, 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651660+0000", 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651660+0000", 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651659+0000", 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651882+0000", 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000222222 2024-04-20T02:50:54.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651896+0000", 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4541e-05 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651905+0000", 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1260000000000004e-06 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.657994+0000", 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0060883889999999996 2024-04-20T02:50:54.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.658012+0000", 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7912000000000001e-05 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678124+0000", 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020112623999999999 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678153+0000", 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8632e-05 2024-04-20T02:50:54.733 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1002 4.0 4:0158e114:::smithi10548823-21:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 7.9407306750000002, 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042474603999999999, 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1002 2024-04-20T02:50:54.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742436+0000", 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:54.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742449+0000", 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3042e-05 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742469+0000", 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0007999999999999e-05 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742481+0000", 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1297999999999999e-05 2024-04-20T02:50:54.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742528+0000", 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7697000000000001e-05 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742598+0000", 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9638e-05 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.784913+0000", 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042314928000000002 2024-04-20T02:50:54.737 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1166 4.9 e35/22 4:99bb9a71:::smithi10548823-15:head v 35'51, mlcod=35'49)", 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:51.353617+0000", 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.3295515099999999, 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026675606000000001, 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.353617+0000", 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.353617+0000", 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.353616+0000", 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354591+0000", 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00097470900000000004 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354609+0000", 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7671000000000001e-05 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354618+0000", 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0420000000000008e-06 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354665+0000", 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7265000000000002e-05 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354689+0000", 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3659000000000002e-05 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.380236+0000", 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025547367000000001 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.380293+0000", 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7093999999999999e-05 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.31814352800000001, 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.742 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.747 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.31088056600000002, 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:50:54.753 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.304938037, 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.30473504499999998, 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.759 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.764 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.30471211799999998, 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:50:54.769 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.29462838400000002, 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:50:54.773 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.27774297599999997, 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:50:54.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:50:54.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:50:54.777 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.261466535, 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:50:54.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:50:54.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:50:54.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:54.781 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:54.783 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:50:54.783 INFO:tasks.rados.rados.0.smithi105.stdout:769: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-20T02:50:54.784 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:54.783+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:54.850 INFO:tasks.rados.rados.0.smithi105.stdout:766: finishing write tid 1 to smithi10548823-27 2024-04-20T02:50:54.850 INFO:tasks.rados.rados.0.smithi105.stdout:766: finishing write tid 2 to smithi10548823-27 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:766: finishing write tid 3 to smithi10548823-27 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:766: finishing write tid 4 to smithi10548823-27 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:766: finishing write tid 5 to smithi10548823-27 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:766: oid 27 updating version 0 to 55 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:766: oid 27 version 55 is already newer than 53 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:766: oid 27 version 55 is already newer than 54 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:766: oid 27 version 55 is already newer than 52 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 55 (ObjNum 103 snap 5 seq_num 103) dirty exists 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:766: left oid 27 (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:768: finishing rollback tid 0 to smithi10548823-6 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:770: got (22) Invalid argument 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:768: finishing rollback tid 1 to smithi10548823-6 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:764: done (7 left) 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:765: done (6 left) 2024-04-20T02:50:54.851 INFO:tasks.rados.rados.0.smithi105.stdout:766: done (5 left) 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:767: done (4 left) 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:769: done (3 left) 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:770: done (2 left) 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:771: done (1 left) 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:772: read oid 38 snap -1 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:772: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:773: read oid 48 snap 4 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:773: expect (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:774: write oid 7 current snap is 5 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:768: finishing rollback tid 2 to smithi10548823-6 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 59 (ObjNum 92 snap 3 seq_num 92) dirty exists 2024-04-20T02:50:54.852 INFO:tasks.rados.rados.0.smithi105.stdout:774: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-04-20T02:50:54.856 INFO:tasks.rados.rados.0.smithi105.stdout:774: writing smithi10548823-7 from 564735 to 1178469 tid 1 2024-04-20T02:50:54.864 INFO:tasks.rados.rados.0.smithi105.stdout:774: writing smithi10548823-7 from 1828021 to 2497055 tid 2 2024-04-20T02:50:54.870 INFO:tasks.rados.rados.0.smithi105.stdout:774: writing smithi10548823-7 from 3134120 to 3764735 tid 3 2024-04-20T02:50:54.872 INFO:tasks.rados.rados.0.smithi105.stdout:768: done (3 left) 2024-04-20T02:50:54.872 INFO:tasks.rados.rados.0.smithi105.stdout:775: snap_remove snap 2 2024-04-20T02:50:54.883 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:54.883+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:54.917 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:54.917 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:54.917 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:54.917 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:54.927 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:50:54.984 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:54.984+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:55.055 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:55.055 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:55.055 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:55.055 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:55.055 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:55.065 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:50:55.084 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:55.085+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:55.185 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:55.185+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(osd.0.15:86 3.3 3:fc8c1f37:::d6eead2e10707cc007893debcb7e3f1d2f0592877eb2179b53fb8b2234678183:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.7832629240000006, 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043953350000000002, 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.196 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411685+0000", 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.763e-06 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411696+0000", 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0164e-05 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411704+0000", 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3599999999999996e-06 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.412554+0000", 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00084986399999999998 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.412635+0000", 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.1284000000000003e-05 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:55.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.412785+0000", 2024-04-20T02:50:55.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015003799999999999 2024-04-20T02:50:55.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.413631+0000", 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00084596599999999997 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.413863+0000", 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000231559 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.455552+0000", 2024-04-20T02:50:55.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041689853999999998 2024-04-20T02:50:55.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.455600+0000", 2024-04-20T02:50:55.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7352000000000003e-05 2024-04-20T02:50:55.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.455635+0000", 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4758999999999997e-05 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:941 4.c 4:392c1e2f:::smithi10548823-44:head [write 460525~690852,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.517365+0000", 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6775789299999992, 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043750506000000002, 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.203 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.204 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.204 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.204 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.204 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 941 2024-04-20T02:50:55.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.204 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517365+0000", 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517365+0000", 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517361+0000", 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517909+0000", 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00054796999999999995 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517931+0000", 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1642999999999999e-05 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517948+0000", 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6461999999999999e-05 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.518008+0000", 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0783999999999999e-05 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.518088+0000", 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9493000000000006e-05 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.518210+0000", 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012213400000000001 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.553216+0000", 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035006056000000001 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.559230+0000", 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060135939999999997 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.560955+0000", 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001725524 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.561052+0000", 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6465999999999994e-05 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.561116+0000", 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4140000000000006e-05 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:942 4.c 4:392c1e2f:::smithi10548823-44:head [write 1622620~730352] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:55.209 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.523837+0000", 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6711067009999994, 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044628446000000002, 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 942 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.523837+0000", 2024-04-20T02:50:55.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.523837+0000", 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.523836+0000", 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.524505+0000", 2024-04-20T02:50:55.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066824199999999999 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.524518+0000", 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3414000000000001e-05 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.524528+0000", 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0020999999999999e-05 2024-04-20T02:50:55.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.534543+0000", 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010014655000000001 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.534645+0000", 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010243600000000001 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.534828+0000", 2024-04-20T02:50:55.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182358 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.559305+0000", 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024477615000000001 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.560919+0000", 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001613511 2024-04-20T02:50:55.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.568281+0000", 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0073622569999999997 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.568403+0000", 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012183700000000001 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.568466+0000", 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3138000000000004e-05 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:973 4.1 4:8efa5349:::smithi10548823-10:head [write 494139~557703,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.565018+0000", 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6299265179999995, 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055306111999999998, 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.216 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 973 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565018+0000", 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565018+0000", 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565016+0000", 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565810+0000", 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00079402900000000002 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565823+0000", 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2568999999999999e-05 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565829+0000", 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5370000000000003e-06 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565866+0000", 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591000000000003e-05 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565927+0000", 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0621000000000001e-05 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.566035+0000", 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010804 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604559+0000", 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038524709999999997 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.612885+0000", 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0083253379999999998 2024-04-20T02:50:55.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620219+0000", 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0073340899999999997 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620273+0000", 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4163999999999998e-05 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620324+0000", 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0934999999999999e-05 2024-04-20T02:50:55.221 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:974 4.1 4:8efa5349:::smithi10548823-10:head [write 1455937~734021] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.572613+0000", 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.622330882, 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047848988000000002, 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.222 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 974 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.572613+0000", 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.572613+0000", 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.572612+0000", 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.573688+0000", 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010766689999999999 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.573702+0000", 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4003e-05 2024-04-20T02:50:55.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.573710+0000", 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0849999999999997e-06 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578846+0000", 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0051359739999999997 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578946+0000", 2024-04-20T02:50:55.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9740000000000004e-05 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.579075+0000", 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000129102 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604563+0000", 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025487644 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.612927+0000", 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0083637120000000006 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620373+0000", 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074460020000000002 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620422+0000", 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9536999999999997e-05 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620462+0000", 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0082e-05 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:975 4.1 4:8efa5349:::smithi10548823-10:head [write 2948041~346098] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.578264+0000", 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6166806729999994, 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042320522999999999, 2024-04-20T02:50:55.228 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 975 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578264+0000", 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578264+0000", 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578262+0000", 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578837+0000", 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00057476800000000002 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578847+0000", 2024-04-20T02:50:55.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9199999999999999e-06 2024-04-20T02:50:55.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578855+0000", 2024-04-20T02:50:55.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9149999999999996e-06 2024-04-20T02:50:55.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.595577+0000", 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016722131000000001 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.595661+0000", 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4189000000000003e-05 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:55.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.595799+0000", 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013843300000000001 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604568+0000", 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087682909999999992 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.613307+0000", 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087388210000000008 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620505+0000", 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0071982119999999998 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620546+0000", 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1325999999999997e-05 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620584+0000", 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7963000000000002e-05 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.234 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1143 4.9 4:95e19a88:::smithi10548823-40:head [tier-promote] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.932855+0000", 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.2620887590000001, 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053775995, 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1143 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932855+0000", 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932855+0000", 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:50:55.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932861+0000", 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5909999999999998e-06 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932878+0000", 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6081000000000001e-05 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932887+0000", 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7410000000000004e-06 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932954+0000", 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6551999999999996e-05 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.933079+0000", 2024-04-20T02:50:55.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012470600000000001 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986631+0000", 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053552884000000002 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.2519671739999998, 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:50:55.239 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:50:55.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:50:55.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.2444888560000003, 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:50:55.245 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1146 4.d 4:b376c20e:::smithi10548823-49:head [write 3437750~125301] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.2399209950000003, 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070707487999999999, 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.251 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1146 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:50:55.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955022+0000", 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955179+0000", 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015657 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955188+0000", 2024-04-20T02:50:55.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.645e-06 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955195+0000", 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2910000000000004e-06 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978329+0000", 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02313401 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978450+0000", 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121438 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978652+0000", 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020163700000000001 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984380+0000", 2024-04-20T02:50:55.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057282510000000002 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991325+0000", 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069446830000000001 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025680+0000", 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034355284 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025709+0000", 2024-04-20T02:50:55.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8781000000000001e-05 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025731+0000", 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1963e-05 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1147 4.d 4:b376c20e:::smithi10548823-49:head [setxattr _header (56) in=7b,truncate 3563051] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.2397426579999999, 2024-04-20T02:50:55.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070713812000000001, 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1147 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955201+0000", 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955204+0000", 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.05e-06 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955207+0000", 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1949999999999999e-06 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955210+0000", 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983351+0000", 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028140978000000001 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983488+0000", 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013647 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983668+0000", 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000180335 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984487+0000", 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081847099999999998 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991393+0000", 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069068569999999998 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025852+0000", 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034458946999999997 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025883+0000", 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0261e-05 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025915+0000", 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2654999999999998e-05 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.2397309569999999, 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:55.263 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:50:55.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:50:55.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:50:55.267 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.81630202299999999, 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.268 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:50:55.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.81626513999999994, 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.81624050699999995, 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:55.281 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:50:55.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.80974709600000006, 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.286 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:50:55.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:50:55.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:50:55.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:50:55.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:50:55.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.292 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.80368307400000005, 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.293 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:50:55.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:50:55.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:50:55.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:50:55.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.79362130900000005, 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.299 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:50:55.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.79292870500000001, 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:50:55.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:50:55.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:50:55.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:50:55.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.79291705099999998, 2024-04-20T02:50:55.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:50:55.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.317 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.317 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:50:55.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:50:55.317 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.317 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.317 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.317 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.317 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.317 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:55.318 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:55.319+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:55.319 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:50:55.367 INFO:tasks.rados.rados.0.smithi105.stdout:775: done (3 left) 2024-04-20T02:50:55.367 INFO:tasks.rados.rados.0.smithi105.stdout:776: delete oid 39 current snap is 5 2024-04-20T02:50:55.367 INFO:tasks.rados.rados.0.smithi105.stdout:773: expect (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:50:55.378 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:50:55.434 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:55.433+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:55.494 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:55.494 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:55.494 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:55.494 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:55.504 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:50:55.519 INFO:tasks.rados.rados.0.smithi105.stdout:772: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:50:55.520 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:55.520+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:55.604 INFO:tasks.rados.rados.0.smithi105.stdout:774: finishing write tid 1 to smithi10548823-7 2024-04-20T02:50:55.604 INFO:tasks.rados.rados.0.smithi105.stdout:774: finishing write tid 2 to smithi10548823-7 2024-04-20T02:50:55.604 INFO:tasks.rados.rados.0.smithi105.stdout:774: finishing write tid 3 to smithi10548823-7 2024-04-20T02:50:55.604 INFO:tasks.rados.rados.0.smithi105.stdout:774: finishing write tid 4 to smithi10548823-7 2024-04-20T02:50:55.604 INFO:tasks.rados.rados.0.smithi105.stdout:774: finishing write tid 5 to smithi10548823-7 2024-04-20T02:50:55.605 INFO:tasks.rados.rados.0.smithi105.stdout:774: oid 7 updating version 0 to 79 2024-04-20T02:50:55.605 INFO:tasks.rados.rados.0.smithi105.stdout:774: oid 7 updating version 79 to 80 2024-04-20T02:50:55.605 INFO:tasks.rados.rados.0.smithi105.stdout:774: oid 7 updating version 80 to 81 2024-04-20T02:50:55.605 INFO:tasks.rados.rados.0.smithi105.stdout:774: oid 7 version 81 is already newer than 78 2024-04-20T02:50:55.605 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 81 (ObjNum 105 snap 5 seq_num 105) dirty exists 2024-04-20T02:50:55.605 INFO:tasks.rados.rados.0.smithi105.stdout:774: left oid 7 (ObjNum 105 snap 5 seq_num 105) 2024-04-20T02:50:55.606 INFO:tasks.rados.rados.0.smithi105.stdout:772: done (3 left) 2024-04-20T02:50:55.606 INFO:tasks.rados.rados.0.smithi105.stdout:773: done (2 left) 2024-04-20T02:50:55.606 INFO:tasks.rados.rados.0.smithi105.stdout:774: done (1 left) 2024-04-20T02:50:55.606 INFO:tasks.rados.rados.0.smithi105.stdout:776: done (0 left) 2024-04-20T02:50:55.606 INFO:tasks.rados.rados.0.smithi105.stdout:777: write oid 30 current snap is 5 2024-04-20T02:50:55.606 INFO:tasks.rados.rados.0.smithi105.stdout:777: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-04-20T02:50:55.609 INFO:tasks.rados.rados.0.smithi105.stdout:777: writing smithi10548823-30 from 598349 to 1078934 tid 1 2024-04-20T02:50:55.615 INFO:tasks.rados.rados.0.smithi105.stdout:777: writing smithi10548823-30 from 1544985 to 2217688 tid 2 2024-04-20T02:50:55.620 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:55.620+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:777: writing smithi10548823-30 from 2905319 to 3387542 tid 3 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:778: set_chunk oid 39 target oid 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:778: set_chunk oid 39 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:779: read oid 4 snap -1 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:779: expect (ObjNum 96 snap 4 seq_num 96) 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:780: read oid 27 snap -1 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:780: expect (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:781: read oid 31 snap -1 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:781: expect (ObjNum 89 snap 2 seq_num 89) 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:782: read oid 22 snap -1 2024-04-20T02:50:55.622 INFO:tasks.rados.rados.0.smithi105.stdout:782: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-20T02:50:55.623 INFO:tasks.rados.rados.0.smithi105.stdout:783: snap_remove snap 4 2024-04-20T02:50:55.636 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:55.636 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:55.636 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:55.636 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:55.636 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:55.648 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:50:55.721 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:55.721+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:55.817 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.081703842, 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.818 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:50:55.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:50:55.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.050414781000001, 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:50:55.824 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.045724325, 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.830 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:50:55.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:50:55.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.038967855999999, 2024-04-20T02:50:55.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.029152345, 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.842 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:50:55.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:50:55.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.028108413, 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.848 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:50:55.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:50:55.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:50:55.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:50:55.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.028087253000001, 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:55.854 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.858 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.820516672, 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:50:55.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.806322027, 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.863 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:50:55.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:50:55.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:50:55.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.80035477, 2024-04-20T02:50:55.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.871 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.791948767000001, 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.688316726, 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:50:55.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:50:55.879 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.457720416000001, 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:50:55.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.430776267000001, 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.884 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:50:55.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:50:55.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.405417999999999, 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:50:55.888 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:50:55.892 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.377277555999999, 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.896 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.339908951, 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.900 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.339612027999999, 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:50:55.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:50:55.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:50:55.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.33452891, 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.906 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.307526351, 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:55.909 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:50:55.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:50:55.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:50:55.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:55.913 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:55.915 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:50:55.915 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:55.915+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:56.016 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:56.015+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:56.043 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:56.043 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:56.044 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:56.044 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:56.055 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:50:56.110 INFO:tasks.rados.rados.0.smithi105.stdout:783: done (6 left) 2024-04-20T02:50:56.110 INFO:tasks.rados.rados.0.smithi105.stdout:784: snap_create 2024-04-20T02:50:56.111 INFO:tasks.rados.rados.0.smithi105.stdout:778: finishing set_chunk to oid 39 2024-04-20T02:50:56.111 INFO:tasks.rados.rados.0.smithi105.stdout:778: got expected ERANGE (src dne) 2024-04-20T02:50:56.111 INFO:tasks.rados.rados.0.smithi105.stdout:782: expect (ObjNum 21 snap 0 seq_num 21) 2024-04-20T02:50:56.137 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:56.137+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:56.192 INFO:tasks.rados.rados.0.smithi105.stdout:779: expect (ObjNum 96 snap 4 seq_num 96) 2024-04-20T02:50:56.216 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:56.217+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:56.226 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:56.226 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:56.226 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:56.226 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:56.227 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:56.237 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:50:56.318 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:56.319+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:56.378 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.803126577, 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.408 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:50:56.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:50:56.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:29 4.8 4:1c68e542:::smithi10548823-8:head [write 517649~526743,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.050251945999999, 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031979782999999998, 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.415 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 29 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356788+0000", 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357534+0000", 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074630400000000004 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357547+0000", 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2697000000000001e-05 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357556+0000", 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8040000000000007e-06 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357617+0000", 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1038999999999996e-05 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357754+0000", 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013660500000000001 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357853+0000", 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9327000000000006e-05 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388575+0000", 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030721678999999998 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388623+0000", 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878999999999997e-05 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388660+0000", 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7002000000000002e-05 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388713+0000", 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2608999999999997e-05 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388769+0000", 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5896999999999997e-05 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:30 4.8 4:1c68e542:::smithi10548823-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.042974388999999, 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033039351000000002, 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.421 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 30 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364066+0000", 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364312+0000", 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000246708 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364328+0000", 2024-04-20T02:50:56.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5376999999999999e-05 2024-04-20T02:50:56.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364339+0000", 2024-04-20T02:50:56.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.092e-05 2024-04-20T02:50:56.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375692+0000", 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011353103999999999 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375805+0000", 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000113414 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375970+0000", 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016508399999999999 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388615+0000", 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012645145 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391156+0000", 2024-04-20T02:50:56.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025406410000000002 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396990+0000", 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0058340390000000001 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397042+0000", 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2454000000000003e-05 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397106+0000", 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3632000000000003e-05 2024-04-20T02:50:56.427 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:31 4.8 4:1c68e542:::smithi10548823-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.042687693, 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032910542000000001, 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:56.428 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 31 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364357+0000", 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1160000000000001e-06 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364363+0000", 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1680000000000001e-06 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364367+0000", 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8750000000000002e-06 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388814+0000", 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024447437999999998 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388914+0000", 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9557999999999998e-05 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389062+0000", 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014799600000000001 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391160+0000", 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020979760000000001 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396038+0000", 2024-04-20T02:50:56.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048784689999999999 2024-04-20T02:50:56.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397155+0000", 2024-04-20T02:50:56.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001116286 2024-04-20T02:50:56.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397208+0000", 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3273000000000003e-05 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397264+0000", 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5962999999999999e-05 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:32 4.8 4:1c68e542:::smithi10548823-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:56.434 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.042668979, 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033121840999999999, 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 32 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:50:56.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364375+0000", 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1149999999999998e-06 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364382+0000", 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4259999999999997e-06 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364387+0000", 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6129999999999997e-06 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389813+0000", 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025426621 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389905+0000", 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2011999999999999e-05 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397299+0000", 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007393897 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397395+0000", 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5293999999999999e-05 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397494+0000", 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9289000000000004e-05 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:50:56.439 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.039260177999999, 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:50:56.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:50:56.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.443 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.925957136999999, 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:56.444 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.449 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:66 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.920599249, 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033365526, 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 66 2024-04-20T02:50:56.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486438+0000", 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486778+0000", 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000339709 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486798+0000", 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0462000000000002e-05 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486812+0000", 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3589e-05 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496889+0000", 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010077477 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496978+0000", 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8205e-05 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:56.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.497125+0000", 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000147665 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518444+0000", 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.021318713 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519407+0000", 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00096312999999999998 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519727+0000", 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031953500000000002 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519764+0000", 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7406000000000001e-05 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519807+0000", 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3402000000000002e-05 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.455 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:67 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2268912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.920214065, 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033307638000000001, 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 67 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.456 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486826+0000", 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:56.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486831+0000", 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0189999999999997e-06 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486839+0000", 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4710000000000001e-06 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486844+0000", 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.57e-06 2024-04-20T02:50:56.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509240+0000", 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022395241 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509338+0000", 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8034000000000004e-05 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509498+0000", 2024-04-20T02:50:56.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015993999999999999 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518471+0000", 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0089732730000000004 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519859+0000", 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013877130000000001 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520042+0000", 2024-04-20T02:50:56.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182981 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520084+0000", 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2518999999999997e-05 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520135+0000", 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0491000000000002e-05 2024-04-20T02:50:56.461 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.920191620000001, 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:56.462 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:50:56.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:50:56.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:145 4.a e29/22 4:5a553643:::smithi10548823-37:head v 29'4, mlcod=0'0)", 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.776812+0000", 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.630228665000001, 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031041178999999999, 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776812+0000", 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776812+0000", 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776812+0000", 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.777003+0000", 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000191375 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.777015+0000", 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2216e-05 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.777024+0000", 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8149999999999995e-06 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.777050+0000", 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5768000000000001e-05 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.777064+0000", 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4581e-05 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807826+0000", 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030761927000000001 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807854+0000", 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7294999999999999e-05 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:161 4.3 e29/27 4:c32272c0:::smithi10548823-41:head v 29'16, mlcod=29'14)", 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.572755529, 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031990785000000001, 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.471 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834285+0000", 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:56.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835168+0000", 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00088287800000000003 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835180+0000", 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2340999999999999e-05 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835187+0000", 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7830000000000001e-06 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835227+0000", 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0754000000000003e-05 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835243+0000", 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5607000000000002e-05 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866234+0000", 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030990891 2024-04-20T02:50:56.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866276+0000", 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2469999999999998e-05 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.566171485, 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:50:56.475 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:50:56.479 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.55580788, 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:50:56.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:50:56.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.483 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.547723401000001, 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:50:56.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:185 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 773122~559572,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:56.488 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.921100+0000", 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.485941235, 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028107052, 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 185 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.921100+0000", 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.921100+0000", 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.921099+0000", 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.921670+0000", 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00057114000000000004 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.921682+0000", 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2632999999999999e-05 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.921690+0000", 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2640000000000002e-06 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.921730+0000", 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0364000000000002e-05 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.921837+0000", 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010699700000000001 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.921934+0000", 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7161e-05 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.948184+0000", 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026249405999999999 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.948338+0000", 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015458900000000001 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.949140+0000", 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00080233300000000002 2024-04-20T02:50:56.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.949172+0000", 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1170999999999997e-05 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.949207+0000", 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5302000000000002e-05 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.494 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:186 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 1992054~651681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.926120+0000", 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.480920859999999, 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028110169000000001, 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 186 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.926120+0000", 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.926120+0000", 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.926119+0000", 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.927029+0000", 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00091080899999999995 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.927042+0000", 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2699e-05 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.927050+0000", 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.1499999999999999e-06 2024-04-20T02:50:56.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.933827+0000", 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.006777033 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.933917+0000", 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0121e-05 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.934060+0000", 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142144 2024-04-20T02:50:56.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.948418+0000", 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014358229 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.951544+0000", 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0031257839999999999 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.954175+0000", 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002631272 2024-04-20T02:50:56.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.954205+0000", 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0574000000000002e-05 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.954230+0000", 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5021999999999999e-05 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:981 4.7 e32/27 4:e9876a63:::smithi10548823-30:head v 32'19, mlcod=30'17)", 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.817670927, 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033092985999999998, 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589369+0000", 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590134+0000", 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076479 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590148+0000", 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3679999999999999e-05 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590157+0000", 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590209+0000", 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2237999999999997e-05 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590235+0000", 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6251e-05 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622430+0000", 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032194953999999998 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622463+0000", 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2780000000000001e-05 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.504 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1230 4.5 e38/22 4:a94dc531:::smithi10548823-32:head v 38'11, mlcod=29'8)", 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.9945255290000001, 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033312785999999997, 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412515+0000", 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412827+0000", 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031200999999999999 2024-04-20T02:50:56.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412835+0000", 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.61e-06 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412842+0000", 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6990000000000003e-06 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412921+0000", 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8380000000000005e-05 2024-04-20T02:50:56.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412944+0000", 2024-04-20T02:50:56.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3609e-05 2024-04-20T02:50:56.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445795+0000", 2024-04-20T02:50:56.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032850279000000003 2024-04-20T02:50:56.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445828+0000", 2024-04-20T02:50:56.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3673000000000001e-05 2024-04-20T02:50:56.509 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1262 4.3 e38/27 4:c07c6daa:::smithi10548823-7:head v 38'78, mlcod=38'75)", 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.858673+0000", 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.5483678240000001, 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027983785000000001, 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:56.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.858673+0000", 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.858673+0000", 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.858671+0000", 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:56.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.859327+0000", 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000655497 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.859343+0000", 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5770999999999999e-05 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.859352+0000", 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2629999999999992e-06 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:56.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.859403+0000", 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0730999999999999e-05 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.859423+0000", 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0302999999999999e-05 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.886612+0000", 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027188677000000001 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:56.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.886657+0000", 2024-04-20T02:50:56.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5383999999999998e-05 2024-04-20T02:50:56.514 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.514 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.514 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.514 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:56.514 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:56.514 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:56.515 INFO:tasks.rados.rados.0.smithi105.stdout:780: expect (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:50:56.516 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:50:56.516 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:56.516+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:56.523 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:50:56.542 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:50:56.561 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:50:56.580 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:50:56.599 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:50:56.616 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:56.617+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:56.618 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:50:56.630 INFO:tasks.rados.rados.0.smithi105.stdout:781: expect (ObjNum 89 snap 2 seq_num 89) 2024-04-20T02:50:56.637 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:50:56.656 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:50:56.658 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:50:56.658 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:50:56.658 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:50:56.658 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:50:56.658 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:50:56.658 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-20T02:50:56.658 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:50:56.658 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:50:56.686 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:56.686 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:56.697 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:50:56.717 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:56.717+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:777: finishing write tid 1 to smithi10548823-30 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:777: finishing write tid 2 to smithi10548823-30 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:777: finishing write tid 3 to smithi10548823-30 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:777: finishing write tid 4 to smithi10548823-30 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:777: finishing write tid 5 to smithi10548823-30 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:777: oid 30 updating version 0 to 24 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:777: oid 30 updating version 24 to 26 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:777: oid 30 updating version 26 to 27 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:777: oid 30 version 27 is already newer than 25 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 27 (ObjNum 107 snap 5 seq_num 107) dirty exists 2024-04-20T02:50:56.779 INFO:tasks.rados.rados.0.smithi105.stdout:777: left oid 30 (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:50:56.818 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:56.819+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:56.824 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:56.824 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:56.834 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:50:56.919 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:56.920+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:50:57.003 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:57.003 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:57.014 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:50:57.054 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:57.055+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:777: done (6 left) 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:778: done (5 left) 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:779: done (4 left) 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:780: done (3 left) 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:781: done (2 left) 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:782: done (1 left) 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:784: done (0 left) 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:785: read oid 45 snap -1 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:785: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:786: read oid 43 snap -1 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:786: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:787: write oid 42 current snap is 6 2024-04-20T02:50:57.116 INFO:tasks.rados.rados.0.smithi105.stdout:787: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-04-20T02:50:57.122 INFO:tasks.rados.rados.0.smithi105.stdout:787: writing smithi10548823-42 from 615156 to 1170990 tid 1 2024-04-20T02:50:57.122 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:57.123+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:57.126 INFO:tasks.rados.rados.0.smithi105.stdout:787: writing smithi10548823-42 from 1603467 to 2019828 tid 2 2024-04-20T02:50:57.135 INFO:tasks.rados.rados.0.smithi105.stdout:787: writing smithi10548823-42 from 2816389 to 3509884 tid 3 2024-04-20T02:50:57.135 INFO:tasks.rados.rados.0.smithi105.stdout:788: delete oid 35 current snap is 6 2024-04-20T02:50:57.135 INFO:tasks.rados.rados.0.smithi105.stdout:786: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:50:57.165 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:57.166 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:57.179 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:50:57.223 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:57.224+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:57.239 INFO:tasks.rados.rados.0.smithi105.stdout:785: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-20T02:50:57.307 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:57.308 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:57.318 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:50:57.330 INFO:tasks.rados.rados.0.smithi105.stdout:787: finishing write tid 1 to smithi10548823-42 2024-04-20T02:50:57.330 INFO:tasks.rados.rados.0.smithi105.stdout:787: finishing write tid 2 to smithi10548823-42 2024-04-20T02:50:57.330 INFO:tasks.rados.rados.0.smithi105.stdout:785: done (3 left) 2024-04-20T02:50:57.330 INFO:tasks.rados.rados.0.smithi105.stdout:786: done (2 left) 2024-04-20T02:50:57.330 INFO:tasks.rados.rados.0.smithi105.stdout:788: done (1 left) 2024-04-20T02:50:57.330 INFO:tasks.rados.rados.0.smithi105.stdout:789: set_chunk oid 35 target oid 2024-04-20T02:50:57.330 INFO:tasks.rados.rados.0.smithi105.stdout:789: set_chunk oid 35 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:50:57.330 INFO:tasks.rados.rados.0.smithi105.stdout:790: delete oid 11 current snap is 6 2024-04-20T02:50:57.331 INFO:tasks.rados.rados.0.smithi105.stdout:787: finishing write tid 3 to smithi10548823-42 2024-04-20T02:50:57.331 INFO:tasks.rados.rados.0.smithi105.stdout:787: finishing write tid 4 to smithi10548823-42 2024-04-20T02:50:57.331 INFO:tasks.rados.rados.0.smithi105.stdout:787: finishing write tid 5 to smithi10548823-42 2024-04-20T02:50:57.331 INFO:tasks.rados.rados.0.smithi105.stdout:787: oid 42 updating version 0 to 20 2024-04-20T02:50:57.331 INFO:tasks.rados.rados.0.smithi105.stdout:787: oid 42 version 20 is already newer than 19 2024-04-20T02:50:57.331 INFO:tasks.rados.rados.0.smithi105.stdout:787: oid 42 updating version 20 to 21 2024-04-20T02:50:57.331 INFO:tasks.rados.rados.0.smithi105.stdout:787: oid 42 version 21 is already newer than 18 2024-04-20T02:50:57.331 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 21 (ObjNum 108 snap 6 seq_num 108) dirty exists 2024-04-20T02:50:57.331 INFO:tasks.rados.rados.0.smithi105.stdout:787: left oid 42 (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:50:57.332 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:57.330+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:57.332 INFO:tasks.rados.rados.0.smithi105.stdout:789: finishing set_chunk to oid 35 2024-04-20T02:50:57.332 INFO:tasks.rados.rados.0.smithi105.stdout:789: got expected ERANGE (src dne) 2024-04-20T02:50:57.333 INFO:tasks.rados.rados.0.smithi105.stdout:787: done (2 left) 2024-04-20T02:50:57.333 INFO:tasks.rados.rados.0.smithi105.stdout:789: done (1 left) 2024-04-20T02:50:57.333 INFO:tasks.rados.rados.0.smithi105.stdout:790: done (0 left) 2024-04-20T02:50:57.333 INFO:tasks.rados.rados.0.smithi105.stdout:791: read oid 42 snap -1 2024-04-20T02:50:57.333 INFO:tasks.rados.rados.0.smithi105.stdout:791: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:50:57.333 INFO:tasks.rados.rados.0.smithi105.stdout:792: read oid 45 snap -1 2024-04-20T02:50:57.333 INFO:tasks.rados.rados.0.smithi105.stdout:792: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-20T02:50:57.333 INFO:tasks.rados.rados.0.smithi105.stdout:793: read oid 9 snap -1 2024-04-20T02:50:57.333 INFO:tasks.rados.rados.0.smithi105.stdout:793: expect (ObjNum 60 snap 0 seq_num 60) 2024-04-20T02:50:57.333 INFO:tasks.rados.rados.0.smithi105.stdout:794: snap_create 2024-04-20T02:50:57.338 INFO:tasks.rados.rados.0.smithi105.stdout:793: expect (ObjNum 60 snap 0 seq_num 60) 2024-04-20T02:50:57.393 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:50:57.437 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:57.437+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:57.456 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:57.457 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:57.467 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:50:57.524 INFO:tasks.rados.rados.0.smithi105.stdout:792: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-20T02:50:57.526 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:57.527+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:57.616 INFO:tasks.rados.rados.0.smithi105.stdout:791: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:50:57.628 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:57.628+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:57.659 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:57.659 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:57.670 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:50:57.728 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:57.729+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:57.829 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:57.829+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:57.837 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:57.838 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:57.848 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:50:57.933 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:57.933+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:58.018 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:58.018 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:58.028 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:50:58.031 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:58.031+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:58.113 INFO:tasks.rados.rados.0.smithi105.stdout:791: done (3 left) 2024-04-20T02:50:58.113 INFO:tasks.rados.rados.0.smithi105.stdout:792: done (2 left) 2024-04-20T02:50:58.113 INFO:tasks.rados.rados.0.smithi105.stdout:793: done (1 left) 2024-04-20T02:50:58.113 INFO:tasks.rados.rados.0.smithi105.stdout:794: done (0 left) 2024-04-20T02:50:58.113 INFO:tasks.rados.rados.0.smithi105.stdout:795: set_chunk oid 15 target oid 2024-04-20T02:50:58.114 INFO:tasks.rados.rados.0.smithi105.stdout:795: set_chunk oid 15 offset: 3091968 length: 625664 target oid smithi10548823- offset: 3091968 snap 5 2024-04-20T02:50:58.114 INFO:tasks.rados.rados.0.smithi105.stdout:796: read oid 48 snap -1 2024-04-20T02:50:58.114 INFO:tasks.rados.rados.0.smithi105.stdout:796: expect (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:50:58.114 INFO:tasks.rados.rados.0.smithi105.stdout:797: set_chunk oid 27 target oid 2024-04-20T02:50:58.117 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:50:58.132 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:58.132+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:58.161 INFO:tasks.rados.rados.0.smithi105.stdout:797: set_chunk oid 27 offset: 416768 length: 2275328 target oid d01560f3235f7a04f505ea81c9973331cb949e3c4259980be274c65137cfdb84 offset: 0 snap -1 2024-04-20T02:50:58.161 INFO:tasks.rados.rados.0.smithi105.stdout:798: rollback oid 47 current snap is 7 2024-04-20T02:50:58.161 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 47 to 5 2024-04-20T02:50:58.161 INFO:tasks.rados.rados.0.smithi105.stdout:795: finishing set_chunk to oid 15 2024-04-20T02:50:58.161 INFO:tasks.rados.rados.0.smithi105.stdout:795: got expected ENOENT (src dne) 2024-04-20T02:50:58.161 INFO:tasks.rados.rados.0.smithi105.stdout:796: expect (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:50:58.205 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:58.205 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:58.216 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:50:58.232 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:58.233+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:58.313 INFO:tasks.rados.rados.0.smithi105.stdout:797: finishing set_chunk to oid 27 2024-04-20T02:50:58.313 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 416768 length: 2275328 2024-04-20T02:50:58.313 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 103 snap 5 seq_num 103) dirty exists 2024-04-20T02:50:58.313 INFO:tasks.rados.rados.0.smithi105.stdout:798: finishing rollback tid 0 to smithi10548823-47 2024-04-20T02:50:58.313 INFO:tasks.rados.rados.0.smithi105.stdout:795: done (3 left) 2024-04-20T02:50:58.313 INFO:tasks.rados.rados.0.smithi105.stdout:796: done (2 left) 2024-04-20T02:50:58.313 INFO:tasks.rados.rados.0.smithi105.stdout:797: done (1 left) 2024-04-20T02:50:58.313 INFO:tasks.rados.rados.0.smithi105.stdout:799: read oid 20 snap -1 2024-04-20T02:50:58.314 INFO:tasks.rados.rados.0.smithi105.stdout:799: expect (ObjNum 94 snap 3 seq_num 94) 2024-04-20T02:50:58.314 INFO:tasks.rados.rados.0.smithi105.stdout:800: write oid 23 current snap is 7 2024-04-20T02:50:58.314 INFO:tasks.rados.rados.0.smithi105.stdout:800: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-04-20T02:50:58.322 INFO:tasks.rados.rados.0.smithi105.stdout:800: writing smithi10548823-23 from 665577 to 1447158 tid 1 2024-04-20T02:50:58.329 INFO:tasks.rados.rados.0.smithi105.stdout:800: writing smithi10548823-23 from 2062560 to 2626248 tid 2 2024-04-20T02:50:58.332 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:50:58.333+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:50:58.337 INFO:tasks.rados.rados.0.smithi105.stdout:800: writing smithi10548823-23 from 3182305 to 3825969 tid 3 2024-04-20T02:50:58.337 INFO:tasks.rados.rados.0.smithi105.stdout:801: delete oid 40 current snap is 7 2024-04-20T02:50:58.337 INFO:tasks.rados.rados.0.smithi105.stdout:798: finishing rollback tid 1 to smithi10548823-47 2024-04-20T02:50:58.338 INFO:tasks.rados.rados.0.smithi105.stdout:798: finishing rollback tid 2 to smithi10548823-47 2024-04-20T02:50:58.338 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 37 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:50:58.338 INFO:tasks.rados.rados.0.smithi105.stdout:799: expect (ObjNum 94 snap 3 seq_num 94) 2024-04-20T02:50:58.405 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:58.406 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:58.416 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:50:58.433 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:50:58.434+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:50:58.472 INFO:tasks.rados.rados.0.smithi105.stdout:800: finishing write tid 1 to smithi10548823-23 2024-04-20T02:50:58.472 INFO:tasks.rados.rados.0.smithi105.stdout:800: finishing write tid 2 to smithi10548823-23 2024-04-20T02:50:58.472 INFO:tasks.rados.rados.0.smithi105.stdout:798: done (3 left) 2024-04-20T02:50:58.472 INFO:tasks.rados.rados.0.smithi105.stdout:799: done (2 left) 2024-04-20T02:50:58.472 INFO:tasks.rados.rados.0.smithi105.stdout:801: done (1 left) 2024-04-20T02:50:58.473 INFO:tasks.rados.rados.0.smithi105.stdout:802: write (excl) oid 32 current snap is 7 2024-04-20T02:50:58.473 INFO:tasks.rados.rados.0.smithi105.stdout:800: finishing write tid 3 to smithi10548823-23 2024-04-20T02:50:58.473 INFO:tasks.rados.rados.0.smithi105.stdout:800: finishing write tid 4 to smithi10548823-23 2024-04-20T02:50:58.473 INFO:tasks.rados.rados.0.smithi105.stdout:802: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-04-20T02:50:58.478 INFO:tasks.rados.rados.0.smithi105.stdout:802: writing smithi10548823-32 from 699191 to 1231270 tid 1 2024-04-20T02:50:58.484 INFO:tasks.rados.rados.0.smithi105.stdout:802: writing smithi10548823-32 from 1779524 to 2346881 tid 2 2024-04-20T02:50:58.491 INFO:tasks.rados.rados.0.smithi105.stdout:802: writing smithi10548823-32 from 2837151 to 3503359 tid 3 2024-04-20T02:50:58.491 INFO:tasks.rados.rados.0.smithi105.stdout:803: write oid 17 current snap is 7 2024-04-20T02:50:58.491 INFO:tasks.rados.rados.0.smithi105.stdout:803: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-04-20T02:50:58.498 INFO:tasks.rados.rados.0.smithi105.stdout:803: writing smithi10548823-17 from 715998 to 1323326 tid 1 2024-04-20T02:50:58.503 INFO:tasks.rados.rados.0.smithi105.stdout:803: writing smithi10548823-17 from 1838006 to 2265374 tid 2 2024-04-20T02:50:58.509 INFO:tasks.rados.rados.0.smithi105.stdout:803: writing smithi10548823-17 from 2864574 to 3342054 tid 3 2024-04-20T02:50:58.513 INFO:tasks.rados.rados.0.smithi105.stdout:803: writing smithi10548823-17 from 3863835 to 3915998 tid 4 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:804: read oid 9 snap 5 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:804: expect (ObjNum 60 snap 0 seq_num 60) 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:805: read oid 35 snap 5 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:805: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:806: tier_evict oid 27 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:806: tier_evict oid 27 snap -1 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:800: finishing write tid 5 to smithi10548823-23 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:800: oid 23 updating version 0 to 29 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:800: oid 23 version 29 is already newer than 28 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:800: oid 23 updating version 29 to 31 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:800: oid 23 version 31 is already newer than 30 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 31 (ObjNum 111 snap 7 seq_num 111) dirty exists 2024-04-20T02:50:58.514 INFO:tasks.rados.rados.0.smithi105.stdout:800: left oid 23 (ObjNum 111 snap 7 seq_num 111) 2024-04-20T02:50:58.515 INFO:tasks.rados.rados.0.smithi105.stdout:800: done (5 left) 2024-04-20T02:50:58.515 INFO:tasks.rados.rados.0.smithi105.stdout:807: tier_promote oid 48 2024-04-20T02:50:58.515 INFO:tasks.rados.rados.0.smithi105.stdout:808: set_chunk oid 4 target oid 2024-04-20T02:50:58.534 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:58.534+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:58.554 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:58.554 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:58.558 INFO:tasks.rados.rados.0.smithi105.stdout:808: set_chunk oid 4 offset: 216576 length: 558080 target oid 08758b900bab559ed43ce3b1f3ebb6b0e6caa3128a430ced566a2cb335ea1e1d offset: 0 snap -1 2024-04-20T02:50:58.558 INFO:tasks.rados.rados.0.smithi105.stdout:809: set_chunk oid 11 target oid 2024-04-20T02:50:58.559 INFO:tasks.rados.rados.0.smithi105.stdout:809: set_chunk oid 11 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:50:58.559 INFO:tasks.rados.rados.0.smithi105.stdout:810: read oid 46 snap -1 2024-04-20T02:50:58.559 INFO:tasks.rados.rados.0.smithi105.stdout:810: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-20T02:50:58.560 INFO:tasks.rados.rados.0.smithi105.stdout:811: write (excl) oid 31 current snap is 7 2024-04-20T02:50:58.560 INFO:tasks.rados.rados.0.smithi105.stdout:811: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-04-20T02:50:58.567 INFO:tasks.rados.rados.0.smithi105.stdout:811: writing smithi10548823-31 from 732805 to 1531735 tid 1 2024-04-20T02:50:58.569 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:50:58.578 INFO:tasks.rados.rados.0.smithi105.stdout:811: writing smithi10548823-31 from 2296488 to 2867514 tid 2 2024-04-20T02:50:58.581 INFO:tasks.rados.rados.0.smithi105.stdout:811: writing smithi10548823-31 from 3291997 to 3697102 tid 3 2024-04-20T02:50:58.582 INFO:tasks.rados.rados.0.smithi105.stdout:812: read oid 39 snap -1 2024-04-20T02:50:58.582 INFO:tasks.rados.rados.0.smithi105.stdout:812: expect deleted 2024-04-20T02:50:58.582 INFO:tasks.rados.rados.0.smithi105.stdout:813: read oid 2 snap 5 2024-04-20T02:50:58.582 INFO:tasks.rados.rados.0.smithi105.stdout:813: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-20T02:50:58.582 INFO:tasks.rados.rados.0.smithi105.stdout:814: read oid 38 snap 5 2024-04-20T02:50:58.582 INFO:tasks.rados.rados.0.smithi105.stdout:814: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:50:58.582 INFO:tasks.rados.rados.0.smithi105.stdout:815: write oid 24 current snap is 7 2024-04-20T02:50:58.582 INFO:tasks.rados.rados.0.smithi105.stdout:815: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-04-20T02:50:58.592 INFO:tasks.rados.rados.0.smithi105.stdout:815: writing smithi10548823-24 from 749612 to 1223791 tid 1 2024-04-20T02:50:58.595 INFO:tasks.rados.rados.0.smithi105.stdout:815: writing smithi10548823-24 from 1954970 to 2386007 tid 2 2024-04-20T02:50:58.606 INFO:tasks.rados.rados.0.smithi105.stdout:815: writing smithi10548823-24 from 2919420 to 3535797 tid 3 2024-04-20T02:50:58.606 INFO:tasks.rados.rados.0.smithi105.stdout:816: write (excl) oid 28 current snap is 7 2024-04-20T02:50:58.606 INFO:tasks.rados.rados.0.smithi105.stdout:816: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-04-20T02:50:58.610 INFO:tasks.rados.rados.0.smithi105.stdout:816: writing smithi10548823-28 from 766419 to 1315847 tid 1 2024-04-20T02:50:58.625 INFO:tasks.rados.rados.0.smithi105.stdout:816: writing smithi10548823-28 from 2013452 to 2588147 tid 2 2024-04-20T02:50:58.625 INFO:tasks.rados.rados.0.smithi105.stdout:816: writing smithi10548823-28 from 3346843 to 3774492 tid 3 2024-04-20T02:50:58.625 INFO:tasks.rados.rados.0.smithi105.stdout:817: read oid 15 snap -1 2024-04-20T02:50:58.625 INFO:tasks.rados.rados.0.smithi105.stdout:817: expect deleted 2024-04-20T02:50:58.625 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:58.625 INFO:tasks.rados.rados.0.smithi105.stdout:804: expect (ObjNum 60 snap 0 seq_num 60) 2024-04-20T02:50:58.635 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:50:58.635+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:50:58.670 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:50:58.735 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:50:58.736+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:50:58.742 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:50:58.742 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:50:58.754 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:50:58.813 INFO:tasks.rados.rados.0.smithi105.stdout:806: got (22) Invalid argument 2024-04-20T02:50:58.813 INFO:tasks.rados.rados.0.smithi105.stdout:805: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-20T02:50:58.816 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:50:58.837 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:58.837+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:50:58.849 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:50:58.868 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:50:58.889 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:50:58.909 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:50:58.915 INFO:tasks.rados.rados.0.smithi105.stdout:802: finishing write tid 1 to smithi10548823-32 2024-04-20T02:50:58.916 INFO:tasks.rados.rados.0.smithi105.stdout:802: finishing write tid 2 to smithi10548823-32 2024-04-20T02:50:58.916 INFO:tasks.rados.rados.0.smithi105.stdout:802: finishing write tid 3 to smithi10548823-32 2024-04-20T02:50:58.916 INFO:tasks.rados.rados.0.smithi105.stdout:802: finishing write tid 4 to smithi10548823-32 2024-04-20T02:50:58.916 INFO:tasks.rados.rados.0.smithi105.stdout:802: finishing write tid 5 to smithi10548823-32 2024-04-20T02:50:58.916 INFO:tasks.rados.rados.0.smithi105.stdout:802: oid 32 updating version 0 to 28 2024-04-20T02:50:58.916 INFO:tasks.rados.rados.0.smithi105.stdout:802: oid 32 version 28 is already newer than 25 2024-04-20T02:50:58.917 INFO:tasks.rados.rados.0.smithi105.stdout:802: oid 32 version 28 is already newer than 26 2024-04-20T02:50:58.917 INFO:tasks.rados.rados.0.smithi105.stdout:802: oid 32 version 28 is already newer than 27 2024-04-20T02:50:58.917 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 28 (ObjNum 113 snap 7 seq_num 113) dirty exists 2024-04-20T02:50:58.917 INFO:tasks.rados.rados.0.smithi105.stdout:802: left oid 32 (ObjNum 113 snap 7 seq_num 113) 2024-04-20T02:50:58.917 INFO:tasks.rados.rados.0.smithi105.stdout:802: done (15 left) 2024-04-20T02:50:58.918 INFO:tasks.rados.rados.0.smithi105.stdout:804: done (14 left) 2024-04-20T02:50:58.918 INFO:tasks.rados.rados.0.smithi105.stdout:805: done (13 left) 2024-04-20T02:50:58.918 INFO:tasks.rados.rados.0.smithi105.stdout:806: done (12 left) 2024-04-20T02:50:58.918 INFO:tasks.rados.rados.0.smithi105.stdout:818: read oid 47 snap -1 2024-04-20T02:50:58.918 INFO:tasks.rados.rados.0.smithi105.stdout:818: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:50:58.918 INFO:tasks.rados.rados.0.smithi105.stdout:819: read oid 13 snap 5 2024-04-20T02:50:58.919 INFO:tasks.rados.rados.0.smithi105.stdout:819: expect (ObjNum 97 snap 4 seq_num 97) 2024-04-20T02:50:58.919 INFO:tasks.rados.rados.0.smithi105.stdout:820: read oid 50 snap -1 2024-04-20T02:50:58.919 INFO:tasks.rados.rados.0.smithi105.stdout:820: expect deleted 2024-04-20T02:50:58.919 INFO:tasks.rados.rados.0.smithi105.stdout:821: read oid 30 snap -1 2024-04-20T02:50:58.919 INFO:tasks.rados.rados.0.smithi105.stdout:821: expect (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:50:58.919 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:58.920 INFO:tasks.rados.rados.0.smithi105.stdout:809: finishing set_chunk to oid 11 2024-04-20T02:50:58.920 INFO:tasks.rados.rados.0.smithi105.stdout:809: got expected ERANGE (src dne) 2024-04-20T02:50:58.920 INFO:tasks.rados.rados.0.smithi105.stdout:809: done (15 left) 2024-04-20T02:50:58.921 INFO:tasks.rados.rados.0.smithi105.stdout:822: write (excl) oid 40 current snap is 7 2024-04-20T02:50:58.921 INFO:tasks.rados.rados.0.smithi105.stdout:810: expect (ObjNum 45 snap 0 seq_num 45) 2024-04-20T02:50:58.934 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:50:58.937 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:58.938+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:58.949 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:58.949 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:50:58.949 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:50:58.949 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:50:58.949 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:58.953 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:50:58.960 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:50:58.974 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:50:58.976 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:50:58.976 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:50:58.976 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:50:58.976 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:50:58.976 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:50:58.976 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:50:58.976 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:50:58.976 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:50:59.012 INFO:tasks.rados.rados.0.smithi105.stdout:803: finishing write tid 1 to smithi10548823-17 2024-04-20T02:50:59.013 INFO:tasks.rados.rados.0.smithi105.stdout:822: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-04-20T02:50:59.020 INFO:tasks.rados.rados.0.smithi105.stdout:822: writing smithi10548823-40 from 783226 to 1407903 tid 1 2024-04-20T02:50:59.027 INFO:tasks.rados.rados.0.smithi105.stdout:822: writing smithi10548823-40 from 2071934 to 2506640 tid 2 2024-04-20T02:50:59.035 INFO:tasks.rados.rados.0.smithi105.stdout:822: writing smithi10548823-40 from 2974266 to 3613187 tid 3 2024-04-20T02:50:59.035 INFO:tasks.rados.rados.0.smithi105.stdout:810: done (15 left) 2024-04-20T02:50:59.035 INFO:tasks.rados.rados.0.smithi105.stdout:823: read oid 9 snap -1 2024-04-20T02:50:59.035 INFO:tasks.rados.rados.0.smithi105.stdout:823: expect (ObjNum 60 snap 0 seq_num 60) 2024-04-20T02:50:59.035 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:50:59.035 INFO:tasks.rados.rados.0.smithi105.stdout:803: finishing write tid 2 to smithi10548823-17 2024-04-20T02:50:59.035 INFO:tasks.rados.rados.0.smithi105.stdout:803: finishing write tid 3 to smithi10548823-17 2024-04-20T02:50:59.035 INFO:tasks.rados.rados.0.smithi105.stdout:803: finishing write tid 4 to smithi10548823-17 2024-04-20T02:50:59.035 INFO:tasks.rados.rados.0.smithi105.stdout:803: finishing write tid 5 to smithi10548823-17 2024-04-20T02:50:59.036 INFO:tasks.rados.rados.0.smithi105.stdout:803: finishing write tid 6 to smithi10548823-17 2024-04-20T02:50:59.036 INFO:tasks.rados.rados.0.smithi105.stdout:803: oid 17 updating version 0 to 39 2024-04-20T02:50:59.036 INFO:tasks.rados.rados.0.smithi105.stdout:803: oid 17 updating version 39 to 42 2024-04-20T02:50:59.036 INFO:tasks.rados.rados.0.smithi105.stdout:803: oid 17 version 42 is already newer than 41 2024-04-20T02:50:59.036 INFO:tasks.rados.rados.0.smithi105.stdout:803: oid 17 updating version 42 to 43 2024-04-20T02:50:59.036 INFO:tasks.rados.rados.0.smithi105.stdout:803: oid 17 version 43 is already newer than 40 2024-04-20T02:50:59.036 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 43 (ObjNum 114 snap 7 seq_num 114) dirty exists 2024-04-20T02:50:59.036 INFO:tasks.rados.rados.0.smithi105.stdout:803: left oid 17 (ObjNum 114 snap 7 seq_num 114) 2024-04-20T02:50:59.036 INFO:tasks.rados.rados.0.smithi105.stdout:814: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:50:59.038 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:59.038+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:59.089 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:50:59.090 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:50:59.090 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-04-20T02:50:59.090 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-04-20T02:50:59.133 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:50:59.133 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:50:59.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:941 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'19, mlcod=30'17)", 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.613571912999999, 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042482077, 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.134 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518290+0000", 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.135 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518988+0000", 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069878099999999999 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519003+0000", 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4756999999999999e-05 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519013+0000", 2024-04-20T02:50:59.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4250000000000004e-06 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519071+0000", 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8628000000000002e-05 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519106+0000", 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4817000000000002e-05 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560724+0000", 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041618334 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560773+0000", 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8937999999999998e-05 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:942 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'20, mlcod=30'17)", 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:59.140 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.594754804000001, 2024-04-20T02:50:59.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030979784999999999, 2024-04-20T02:50:59.141 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.141 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.141 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.141 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:59.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537107+0000", 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538354+0000", 2024-04-20T02:50:59.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001247232 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538381+0000", 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7008999999999998e-05 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538395+0000", 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4093999999999999e-05 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538476+0000", 2024-04-20T02:50:59.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1045999999999996e-05 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538505+0000", 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8969999999999999e-05 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568053+0000", 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029547370999999999 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568088+0000", 2024-04-20T02:50:59.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5540000000000002e-05 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.588123577999999, 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.145 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:50:59.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:59.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:50:59.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:50:59.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:50:59.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:50:59.151 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.151 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.151 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.152 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.152 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:59.152 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.581902504, 2024-04-20T02:50:59.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:50:59.152 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.152 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.152 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.153 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.153 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.153 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:50:59.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.158 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.573502299999999, 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.159 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:50:59.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:50:59.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:50:59.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:50:59.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:50:59.164 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:971 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (56) in=7b,truncate 3277332] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.573184231000001, 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051333776999999997, 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.165 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 971 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558678+0000", 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558686+0000", 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7330000000000003e-06 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558693+0000", 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.866e-06 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558698+0000", 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.49e-06 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594685+0000", 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035986753000000003 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594792+0000", 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107129 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594958+0000", 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000165429 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595721+0000", 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076360100000000004 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.596242+0000", 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052083800000000003 2024-04-20T02:50:59.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609915+0000", 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013673431 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609958+0000", 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2759000000000002e-05 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610013+0000", 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4421e-05 2024-04-20T02:50:59.170 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.573159655, 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.171 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.175 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop_reply(client.4431.0:968 4.0 e32/22 ondisk, result = 0)", 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.56571937, 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028443566999999999, 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566143+0000", 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566152+0000", 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1759999999999999e-06 2024-04-20T02:50:59.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566163+0000", 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0852999999999999e-05 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566172+0000", 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2229999999999996e-06 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594564+0000", 2024-04-20T02:50:59.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028392473000000001 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594574+0000", 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5899999999999997e-06 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594587+0000", 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3454e-05 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:986 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'5, mlcod=29'3)", 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.495658613, 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.040291053, 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636467+0000", 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026249800000000001 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636478+0000", 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1762e-05 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636493+0000", 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4826e-05 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636546+0000", 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2638e-05 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636570+0000", 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4235000000000001e-05 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676457+0000", 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039887226999999997 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676496+0000", 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8281000000000002e-05 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1002 4.0 4:0158e114:::smithi10548823-21:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.389424799, 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042474603999999999, 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.184 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1002 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742436+0000", 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742449+0000", 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3042e-05 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742469+0000", 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0007999999999999e-05 2024-04-20T02:50:59.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742481+0000", 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1297999999999999e-05 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742528+0000", 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7697000000000001e-05 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742598+0000", 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9638e-05 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.784913+0000", 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042314928000000002 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.188 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1166 4.9 e35/22 4:99bb9a71:::smithi10548823-15:head v 35'51, mlcod=35'49)", 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:51.353617+0000", 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 7.7782456340000001, 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026675606000000001, 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.353617+0000", 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.353617+0000", 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.353616+0000", 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354591+0000", 2024-04-20T02:50:59.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00097470900000000004 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354609+0000", 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7671000000000001e-05 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354618+0000", 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0420000000000008e-06 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354665+0000", 2024-04-20T02:50:59.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7265000000000002e-05 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354689+0000", 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3659000000000002e-05 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.380236+0000", 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025547367000000001 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.380293+0000", 2024-04-20T02:50:59.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7093999999999999e-05 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.7668376519999995, 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:50:59.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:50:59.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:50:59.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:50:59.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.75957469, 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.199 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.204 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.7536321609999996, 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.205 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:50:59.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.210 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.7534291690000003, 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:50:59.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:50:59.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:50:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:50:59.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:50:59.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.7534062419999996, 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:50:59.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:50:59.221 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.7433225080000003, 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:50:59.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.7264371000000001, 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.229 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.7101606589999996, 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:50:59.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:50:59.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1397 4.9 e42/22 4:95e19a88:::smithi10548823-40:head v 42'88, mlcod=42'86)", 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:59.023037+0000", 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.108825767, 2024-04-20T02:50:59.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038292596999999998, 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.023037+0000", 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.023037+0000", 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.023036+0000", 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.024064+0000", 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010281069999999999 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.024099+0000", 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4712000000000002e-05 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.024110+0000", 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0927999999999999e-05 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.024157+0000", 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7162e-05 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.024177+0000", 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9942e-05 2024-04-20T02:50:59.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.061292+0000", 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037114644000000002 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.061330+0000", 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8282999999999999e-05 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:50:59.238 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:50:59.239 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:50:59.239 INFO:tasks.rados.rados.0.smithi105.stdout:808: finishing set_chunk to oid 4 2024-04-20T02:50:59.240 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 70 (ObjNum 96 snap 4 seq_num 96) dirty exists 2024-04-20T02:50:59.240 INFO:tasks.rados.rados.0.smithi105.stdout:813: expect (ObjNum 1 snap 0 seq_num 1) 2024-04-20T02:50:59.241 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:50:59.242 INFO:tasks.rados.rados.0.smithi105.stdout:807: got (0) Success 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 25 (ObjNum 82 snap 1 seq_num 82) dirty exists 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:811: finishing write tid 1 to smithi10548823-31 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:811: finishing write tid 2 to smithi10548823-31 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:811: finishing write tid 3 to smithi10548823-31 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:811: finishing write tid 4 to smithi10548823-31 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:803: done (15 left) 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:807: done (14 left) 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:808: done (13 left) 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:812: done (12 left) 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:813: done (11 left) 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:814: done (10 left) 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:824: write oid 46 current snap is 7 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:811: finishing write tid 5 to smithi10548823-31 2024-04-20T02:50:59.243 INFO:tasks.rados.rados.0.smithi105.stdout:811: oid 31 updating version 0 to 39 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:811: oid 31 version 39 is already newer than 38 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:811: oid 31 version 39 is already newer than 37 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:811: oid 31 version 39 is already newer than 36 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 39 (ObjNum 115 snap 7 seq_num 115) dirty exists 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:811: left oid 31 (ObjNum 115 snap 7 seq_num 115) 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:815: finishing write tid 1 to smithi10548823-24 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:815: finishing write tid 2 to smithi10548823-24 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:815: finishing write tid 3 to smithi10548823-24 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:815: finishing write tid 4 to smithi10548823-24 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:816: finishing write tid 1 to smithi10548823-28 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:824: seq_num 119 ranges {400033=699926,1614063=385970} 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:824: writing smithi10548823-46 from 400033 to 1099959 tid 1 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:824: writing smithi10548823-46 from 1614063 to 2000033 tid 2 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:811: done (10 left) 2024-04-20T02:50:59.244 INFO:tasks.rados.rados.0.smithi105.stdout:825: delete oid 38 current snap is 7 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:815: finishing write tid 5 to smithi10548823-24 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:815: oid 24 updating version 0 to 84 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:815: oid 24 updating version 84 to 86 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:815: oid 24 updating version 86 to 87 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:815: oid 24 version 87 is already newer than 85 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 87 (ObjNum 116 snap 7 seq_num 116) dirty exists 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:815: left oid 24 (ObjNum 116 snap 7 seq_num 116) 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:816: finishing write tid 2 to smithi10548823-28 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:816: finishing write tid 3 to smithi10548823-28 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:816: finishing write tid 4 to smithi10548823-28 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:816: finishing write tid 5 to smithi10548823-28 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:816: oid 28 updating version 0 to 61 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:816: oid 28 version 61 is already newer than 60 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:816: oid 28 updating version 61 to 62 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:816: oid 28 version 62 is already newer than 59 2024-04-20T02:50:59.245 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 62 (ObjNum 117 snap 7 seq_num 117) dirty exists 2024-04-20T02:50:59.246 INFO:tasks.rados.rados.0.smithi105.stdout:816: left oid 28 (ObjNum 117 snap 7 seq_num 117) 2024-04-20T02:50:59.246 INFO:tasks.rados.rados.0.smithi105.stdout:821: expect (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:50:59.246 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:50:59.240+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:50:59.341 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:59.341+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:59.375 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:59.375 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:59.375 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:59.375 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:59.386 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:50:59.442 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:59.442+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:50:59.461 INFO:tasks.ceph.mon.a.smithi105.stderr:2024-04-20T02:50:59.461+0000 7f7f0b14c700 -1 mon.a@0(leader).osd e42 definitely_dead 0 2024-04-20T02:50:59.503 INFO:tasks.rados.rados.0.smithi105.stdout:818: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:50:59.517 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:59.517 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:59.517 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:50:59.517 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:50:59.517 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:59.527 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:50:59.543 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:59.543+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:59.594 INFO:tasks.rados.rados.0.smithi105.stdout:819: expect (ObjNum 97 snap 4 seq_num 97) 2024-04-20T02:50:59.643 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:50:59.644+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:50:59.699 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:59.699 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:50:59.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:50:59.699 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:50:59.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.699 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(osd.0.15:86 3.3 3:fc8c1f37:::d6eead2e10707cc007893debcb7e3f1d2f0592877eb2179b53fb8b2234678183:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.699 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:59.699 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.286011837, 2024-04-20T02:50:59.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043953350000000002, 2024-04-20T02:50:59.699 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411681+0000", 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411685+0000", 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.763e-06 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411696+0000", 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0164e-05 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.411704+0000", 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3599999999999996e-06 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.412554+0000", 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00084986399999999998 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.412635+0000", 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.1284000000000003e-05 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:59.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.412785+0000", 2024-04-20T02:50:59.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015003799999999999 2024-04-20T02:50:59.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.413631+0000", 2024-04-20T02:50:59.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00084596599999999997 2024-04-20T02:50:59.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.413863+0000", 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000231559 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.455552+0000", 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041689853999999998 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.455600+0000", 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7352000000000003e-05 2024-04-20T02:50:59.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.455635+0000", 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4758999999999997e-05 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:941 4.c 4:392c1e2f:::smithi10548823-44:head [write 460525~690852,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.517365+0000", 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.180327843000001, 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043750506000000002, 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.705 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 941 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517365+0000", 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517365+0000", 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517361+0000", 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517909+0000", 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00054796999999999995 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517931+0000", 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1642999999999999e-05 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.517948+0000", 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6461999999999999e-05 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.518008+0000", 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0783999999999999e-05 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.518088+0000", 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9493000000000006e-05 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.518210+0000", 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012213400000000001 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.553216+0000", 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035006056000000001 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.559230+0000", 2024-04-20T02:50:59.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060135939999999997 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.560955+0000", 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001725524 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.561052+0000", 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6465999999999994e-05 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.561116+0000", 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4140000000000006e-05 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:942 4.c 4:392c1e2f:::smithi10548823-44:head [write 1622620~730352] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.523837+0000", 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.173855614000001, 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044628446000000002, 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.711 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 942 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.523837+0000", 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.523837+0000", 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.523836+0000", 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.524505+0000", 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066824199999999999 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.524518+0000", 2024-04-20T02:50:59.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3414000000000001e-05 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.524528+0000", 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0020999999999999e-05 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.534543+0000", 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010014655000000001 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.534645+0000", 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010243600000000001 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.534828+0000", 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182358 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.559305+0000", 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024477615000000001 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.560919+0000", 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001613511 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.568281+0000", 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0073622569999999997 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.568403+0000", 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012183700000000001 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.568466+0000", 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3138000000000004e-05 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:973 4.1 4:8efa5349:::smithi10548823-10:head [write 494139~557703,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.565018+0000", 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.132675430999999, 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055306111999999998, 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.717 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 973 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565018+0000", 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565018+0000", 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565016+0000", 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565810+0000", 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00079402900000000002 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565823+0000", 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2568999999999999e-05 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565829+0000", 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5370000000000003e-06 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565866+0000", 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591000000000003e-05 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.565927+0000", 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0621000000000001e-05 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.566035+0000", 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010804 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604559+0000", 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038524709999999997 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.612885+0000", 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0083253379999999998 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620219+0000", 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0073340899999999997 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620273+0000", 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4163999999999998e-05 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620324+0000", 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0934999999999999e-05 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.722 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:974 4.1 4:8efa5349:::smithi10548823-10:head [write 1455937~734021] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.572613+0000", 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.125079795, 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047848988000000002, 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 974 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.723 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.572613+0000", 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.572613+0000", 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.572612+0000", 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:59.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.573688+0000", 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010766689999999999 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.573702+0000", 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4003e-05 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.573710+0000", 2024-04-20T02:50:59.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0849999999999997e-06 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578846+0000", 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0051359739999999997 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578946+0000", 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9740000000000004e-05 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:59.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.579075+0000", 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000129102 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604563+0000", 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025487644 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.612927+0000", 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0083637120000000006 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620373+0000", 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074460020000000002 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620422+0000", 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9536999999999997e-05 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620462+0000", 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0082e-05 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:975 4.1 4:8efa5349:::smithi10548823-10:head [write 2948041~346098] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.578264+0000", 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.119429586000001, 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042320522999999999, 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 975 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578264+0000", 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578264+0000", 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578262+0000", 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:59.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578837+0000", 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00057476800000000002 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578847+0000", 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9199999999999999e-06 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.578855+0000", 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9149999999999996e-06 2024-04-20T02:50:59.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.595577+0000", 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016722131000000001 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.595661+0000", 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4189000000000003e-05 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.595799+0000", 2024-04-20T02:50:59.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013843300000000001 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.604568+0000", 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087682909999999992 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.613307+0000", 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087388210000000008 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620505+0000", 2024-04-20T02:50:59.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0071982119999999998 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620546+0000", 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1325999999999997e-05 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.620584+0000", 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7963000000000002e-05 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1143 4.9 4:95e19a88:::smithi10548823-40:head [tier-promote] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.932855+0000", 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.7648376720000005, 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053775995, 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1143 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932855+0000", 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932855+0000", 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932861+0000", 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5909999999999998e-06 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932878+0000", 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6081000000000001e-05 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932887+0000", 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7410000000000004e-06 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.932954+0000", 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6551999999999996e-05 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.933079+0000", 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012470600000000001 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986631+0000", 2024-04-20T02:50:59.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053552884000000002 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.7547160870000003, 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.739 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:50:59.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:50:59.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.7472377689999998, 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.745 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:50:59.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:50:59.750 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1146 4.d 4:b376c20e:::smithi10548823-49:head [write 3437750~125301] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.7426699079999999, 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070707487999999999, 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1146 2024-04-20T02:50:59.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955022+0000", 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955179+0000", 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015657 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955188+0000", 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.645e-06 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955195+0000", 2024-04-20T02:50:59.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2910000000000004e-06 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978329+0000", 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02313401 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978450+0000", 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121438 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:59.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978652+0000", 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020163700000000001 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984380+0000", 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057282510000000002 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991325+0000", 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069446830000000001 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025680+0000", 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034355284 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025709+0000", 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8781000000000001e-05 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025731+0000", 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1963e-05 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1147 4.d 4:b376c20e:::smithi10548823-49:head [setxattr _header (56) in=7b,truncate 3563051] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.7424915710000004, 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070713812000000001, 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1147 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955201+0000", 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955204+0000", 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.05e-06 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955207+0000", 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1949999999999999e-06 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955210+0000", 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983351+0000", 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028140978000000001 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983488+0000", 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013647 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983668+0000", 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000180335 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984487+0000", 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081847099999999998 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991393+0000", 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069068569999999998 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025852+0000", 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034458946999999997 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025883+0000", 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0261e-05 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025915+0000", 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2654999999999998e-05 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:50:59.762 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.7424798700000004, 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:59.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:50:59.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:50:59.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:50:59.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.319050936, 2024-04-20T02:50:59.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:50:59.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:50:59.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:50:59.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:50:59.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.3190140530000001, 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.774 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:50:59.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:50:59.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.780 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.3189894200000003, 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:50:59.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:50:59.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.785 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.3124960090000002, 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.3064319869999999, 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.792 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.797 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.2963702220000002, 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:50:59.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:50:59.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:50:59.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:50:59.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:50:59.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.295677618, 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:50:59.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:50:59.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:50:59.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.809 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.2956659640000003, 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:50:59.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:50:59.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:50:59.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:50:59.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:50:59.814 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:59.815 INFO:tasks.rados.rados.0.smithi105.stdout:823: expect (ObjNum 60 snap 0 seq_num 60) 2024-04-20T02:50:59.816 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:50:59.816 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:50:59.816+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:50:59.917 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:50:59.917+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:50:59.935 INFO:tasks.rados.rados.0.smithi105.stdout:822: finishing write tid 1 to smithi10548823-40 2024-04-20T02:50:59.935 INFO:tasks.rados.rados.0.smithi105.stdout:822: finishing write tid 2 to smithi10548823-40 2024-04-20T02:50:59.935 INFO:tasks.rados.rados.0.smithi105.stdout:822: finishing write tid 3 to smithi10548823-40 2024-04-20T02:50:59.935 INFO:tasks.rados.rados.0.smithi105.stdout:822: finishing write tid 4 to smithi10548823-40 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:822: finishing write tid 5 to smithi10548823-40 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:822: oid 40 updating version 0 to 90 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:822: oid 40 version 90 is already newer than 88 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:822: oid 40 version 90 is already newer than 89 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:822: oid 40 updating version 90 to 91 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 91 (ObjNum 118 snap 7 seq_num 118) dirty exists 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:822: left oid 40 (ObjNum 118 snap 7 seq_num 118) 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:815: done (10 left) 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:816: done (9 left) 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:817: done (8 left) 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:818: done (7 left) 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:819: done (6 left) 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:820: done (5 left) 2024-04-20T02:50:59.936 INFO:tasks.rados.rados.0.smithi105.stdout:821: done (4 left) 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:822: done (3 left) 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:823: done (2 left) 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:825: done (1 left) 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:826: tier_evict oid 29 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:826: tier_evict oid 29 snap -1 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:824: finishing write tid 1 to smithi10548823-46 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:824: finishing write tid 2 to smithi10548823-46 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:824: finishing write tid 3 to smithi10548823-46 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:824: finishing write tid 4 to smithi10548823-46 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:824: oid 46 updating version 0 to 32 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:824: oid 46 version 32 is already newer than 31 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:824: oid 46 updating version 32 to 33 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 33 (ObjNum 119 snap 7 seq_num 119) dirty exists 2024-04-20T02:50:59.937 INFO:tasks.rados.rados.0.smithi105.stdout:824: left oid 46 (ObjNum 119 snap 7 seq_num 119) 2024-04-20T02:50:59.938 INFO:tasks.rados.rados.0.smithi105.stdout:824: done (1 left) 2024-04-20T02:50:59.938 INFO:tasks.rados.rados.0.smithi105.stdout:827: read oid 14 snap -1 2024-04-20T02:50:59.938 INFO:tasks.rados.rados.0.smithi105.stdout:827: expect (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:50:59.938 INFO:tasks.rados.rados.0.smithi105.stdout:828: rollback oid 49 current snap is 7 2024-04-20T02:50:59.938 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 49 to 6 2024-04-20T02:50:59.938 INFO:tasks.rados.rados.0.smithi105.stdout:829: set_chunk oid 21 target oid 2024-04-20T02:50:59.942 INFO:tasks.rados.rados.0.smithi105.stdout:829: set_chunk oid 21 offset: 260608 length: 245248 target oid 18e909acedd84d15199e4a3d0e4b313826ba3b3d6d5ff88b0c9d99956a27f398 offset: 0 snap -1 2024-04-20T02:50:59.942 INFO:tasks.rados.rados.0.smithi105.stdout:830: rollback oid 9 current snap is 7 2024-04-20T02:50:59.943 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 9 to 6 2024-04-20T02:50:59.943 INFO:tasks.rados.rados.0.smithi105.stdout:826: got (22) Invalid argument 2024-04-20T02:50:59.943 INFO:tasks.rados.rados.0.smithi105.stdout:826: done (4 left) 2024-04-20T02:50:59.943 INFO:tasks.rados.rados.0.smithi105.stdout:831: tier_evict oid 6 2024-04-20T02:50:59.943 INFO:tasks.rados.rados.0.smithi105.stdout:831: tier_evict oid 6 snap -1 2024-04-20T02:50:59.943 INFO:tasks.rados.rados.0.smithi105.stdout:828: finishing rollback tid 0 to smithi10548823-49 2024-04-20T02:50:59.943 INFO:tasks.rados.rados.0.smithi105.stdout:828: finishing rollback tid 1 to smithi10548823-49 2024-04-20T02:50:59.943 INFO:tasks.rados.rados.0.smithi105.stdout:828: finishing rollback tid 2 to smithi10548823-49 2024-04-20T02:50:59.943 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 43 (ObjNum 93 snap 3 seq_num 93) dirty exists 2024-04-20T02:50:59.943 INFO:tasks.rados.rados.0.smithi105.stdout:827: expect (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:50:59.988 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:50:59.989 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:50:59.989 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:50:59.989 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:50:59.998 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:00.018 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:00.018+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:00.107 INFO:tasks.rados.rados.0.smithi105.stdout:829: finishing set_chunk to oid 21 2024-04-20T02:51:00.107 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk returned error code -95 offset: 260608 length: 245248 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:831: got (22) Invalid argument 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:830: finishing rollback tid 0 to smithi10548823-9 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:830: finishing rollback tid 1 to smithi10548823-9 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:827: done (4 left) 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:828: done (3 left) 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:829: done (2 left) 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:831: done (1 left) 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:832: tier_flush oid 44 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:832: tier_flush oid 44 snap -1 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:830: finishing rollback tid 2 to smithi10548823-9 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 96 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:830: done (1 left) 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:833: write oid 28 current snap is 7 2024-04-20T02:51:00.108 INFO:tasks.rados.rados.0.smithi105.stdout:833: seq_num 121 ranges {433647=450424,1331027=582033} 2024-04-20T02:51:00.111 INFO:tasks.rados.rados.0.smithi105.stdout:833: writing smithi10548823-28 from 433647 to 884071 tid 1 2024-04-20T02:51:00.119 INFO:tasks.rados.rados.0.smithi105.stdout:833: writing smithi10548823-28 from 1331027 to 1913060 tid 2 2024-04-20T02:51:00.119 INFO:tasks.rados.rados.0.smithi105.stdout:834: set_chunk oid 35 target oid 2024-04-20T02:51:00.119 INFO:tasks.rados.rados.0.smithi105.stdout:834: set_chunk oid 35 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:51:00.119 INFO:tasks.rados.rados.0.smithi105.stdout:835: read oid 19 snap -1 2024-04-20T02:51:00.119 INFO:tasks.rados.rados.0.smithi105.stdout:835: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:00.119 INFO:tasks.rados.rados.0.smithi105.stdout:836: write oid 5 current snap is 7 2024-04-20T02:51:00.119 INFO:tasks.rados.rados.0.smithi105.stdout:836: seq_num 122 ranges {450454=642026,1505862=544592} 2024-04-20T02:51:00.120 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:00.119+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:00.125 INFO:tasks.rados.rados.0.smithi105.stdout:836: writing smithi10548823-5 from 450454 to 1092480 tid 1 2024-04-20T02:51:00.131 INFO:tasks.rados.rados.0.smithi105.stdout:836: writing smithi10548823-5 from 1505862 to 2050454 tid 2 2024-04-20T02:51:00.131 INFO:tasks.rados.rados.0.smithi105.stdout:837: write oid 17 current snap is 7 2024-04-20T02:51:00.131 INFO:tasks.rados.rados.0.smithi105.stdout:837: seq_num 123 ranges {467261=717275,1847991=219270} 2024-04-20T02:51:00.133 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:51:00.143 INFO:tasks.rados.rados.0.smithi105.stdout:837: writing smithi10548823-17 from 467261 to 1184536 tid 1 2024-04-20T02:51:00.144 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:00.144 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:00.144 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:00.144 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:00.144 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:00.146 INFO:tasks.rados.rados.0.smithi105.stdout:837: writing smithi10548823-17 from 1847991 to 2067261 tid 2 2024-04-20T02:51:00.146 INFO:tasks.rados.rados.0.smithi105.stdout:838: tier_evict oid 31 2024-04-20T02:51:00.146 INFO:tasks.rados.rados.0.smithi105.stdout:838: tier_evict oid 31 snap -1 2024-04-20T02:51:00.146 INFO:tasks.rados.rados.0.smithi105.stdout:834: finishing set_chunk to oid 35 2024-04-20T02:51:00.147 INFO:tasks.rados.rados.0.smithi105.stdout:834: got expected ERANGE (src dne) 2024-04-20T02:51:00.147 INFO:tasks.rados.rados.0.smithi105.stdout:834: done (6 left) 2024-04-20T02:51:00.147 INFO:tasks.rados.rados.0.smithi105.stdout:839: snap_create 2024-04-20T02:51:00.147 INFO:tasks.rados.rados.0.smithi105.stdout:838: got (22) Invalid argument 2024-04-20T02:51:00.155 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:00.167 INFO:tasks.rados.rados.0.smithi105.stdout:833: finishing write tid 1 to smithi10548823-28 2024-04-20T02:51:00.173 INFO:tasks.rados.rados.0.smithi105.stdout:835: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:00.219 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:00.219+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:00.296 INFO:tasks.rados.rados.0.smithi105.stdout:837: finishing write tid 1 to smithi10548823-17 2024-04-20T02:51:00.296 INFO:tasks.rados.rados.0.smithi105.stdout:837: finishing write tid 2 to smithi10548823-17 2024-04-20T02:51:00.296 INFO:tasks.rados.rados.0.smithi105.stdout:837: finishing write tid 3 to smithi10548823-17 2024-04-20T02:51:00.296 INFO:tasks.rados.rados.0.smithi105.stdout:837: finishing write tid 4 to smithi10548823-17 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:837: oid 17 updating version 0 to 45 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:837: oid 17 version 45 is already newer than 44 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:837: oid 17 updating version 45 to 46 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 46 (ObjNum 123 snap 7 seq_num 123) dirty exists 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:837: left oid 17 (ObjNum 123 snap 7 seq_num 123) 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:833: finishing write tid 2 to smithi10548823-28 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:833: finishing write tid 3 to smithi10548823-28 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:833: finishing write tid 4 to smithi10548823-28 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:833: oid 28 updating version 0 to 64 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:833: oid 28 version 64 is already newer than 63 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:833: oid 28 updating version 64 to 65 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 65 (ObjNum 121 snap 7 seq_num 121) dirty exists 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:833: left oid 28 (ObjNum 121 snap 7 seq_num 121) 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:832: got (0) Success 2024-04-20T02:51:00.297 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 75 snap 1 seq_num 75) dirty exists 2024-04-20T02:51:00.298 INFO:tasks.rados.rados.0.smithi105.stdout:836: finishing write tid 1 to smithi10548823-5 2024-04-20T02:51:00.298 INFO:tasks.rados.rados.0.smithi105.stdout:836: finishing write tid 2 to smithi10548823-5 2024-04-20T02:51:00.298 INFO:tasks.rados.rados.0.smithi105.stdout:836: finishing write tid 3 to smithi10548823-5 2024-04-20T02:51:00.298 INFO:tasks.rados.rados.0.smithi105.stdout:836: finishing write tid 4 to smithi10548823-5 2024-04-20T02:51:00.298 INFO:tasks.rados.rados.0.smithi105.stdout:836: oid 5 updating version 0 to 68 2024-04-20T02:51:00.298 INFO:tasks.rados.rados.0.smithi105.stdout:836: oid 5 updating version 68 to 69 2024-04-20T02:51:00.298 INFO:tasks.rados.rados.0.smithi105.stdout:836: oid 5 version 69 is already newer than 67 2024-04-20T02:51:00.298 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 69 (ObjNum 122 snap 7 seq_num 122) dirty exists 2024-04-20T02:51:00.298 INFO:tasks.rados.rados.0.smithi105.stdout:836: left oid 5 (ObjNum 122 snap 7 seq_num 122) 2024-04-20T02:51:00.320 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:00.320+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:00.326 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:00.327 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:00.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:00.327 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:00.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.327 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.327 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:00.327 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.590638554000002, 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:51:00.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:51:00.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:51:00.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:51:00.334 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.334 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.334 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.334 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.334 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:00.334 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.559349492999999, 2024-04-20T02:51:00.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:51:00.335 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.335 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:00.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:51:00.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:00.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:51:00.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:51:00.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:51:00.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.554659037, 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.342 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:51:00.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:51:00.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:51:00.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:51:00.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:51:00.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:51:00.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:51:00.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:00.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.547902568000001, 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.349 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:51:00.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.538087056999998, 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:51:00.354 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:51:00.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:51:00.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:51:00.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:51:00.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.537043125, 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:51:00.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:51:00.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:51:00.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:51:00.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:51:00.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:51:00.365 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.537021965000001, 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:51:00.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:00.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:51:00.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:51:00.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.370 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.329451383999999, 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:00.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:51:00.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:51:00.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.315256738999999, 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.309289482000001, 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.379 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.382 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.300883478999999, 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.386 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.197251437999999, 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.966655127999999, 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.392 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:51:00.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.939710979000001, 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:51:00.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.914352711999999, 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:51:00.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:51:00.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.886212268000001, 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:00.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:51:00.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:51:00.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.406 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.848843663, 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:00.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:51:00.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:51:00.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.410 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.84854674, 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:00.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:51:00.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:51:00.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.414 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.843463622000002, 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:00.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:51:00.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:00.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:51:00.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:51:00.419 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.816461062999998, 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:51:00.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:51:00.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:51:00.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:51:00.426 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.426 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.426 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.426 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.426 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.426 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:00.427 INFO:tasks.rados.rados.0.smithi105.stdout:832: done (6 left) 2024-04-20T02:51:00.427 INFO:tasks.rados.rados.0.smithi105.stdout:833: done (5 left) 2024-04-20T02:51:00.427 INFO:tasks.rados.rados.0.smithi105.stdout:835: done (4 left) 2024-04-20T02:51:00.427 INFO:tasks.rados.rados.0.smithi105.stdout:836: done (3 left) 2024-04-20T02:51:00.427 INFO:tasks.rados.rados.0.smithi105.stdout:837: done (2 left) 2024-04-20T02:51:00.428 INFO:tasks.rados.rados.0.smithi105.stdout:838: done (1 left) 2024-04-20T02:51:00.428 INFO:tasks.rados.rados.0.smithi105.stdout:839: done (0 left) 2024-04-20T02:51:00.428 INFO:tasks.rados.rados.0.smithi105.stdout:840: snap_create 2024-04-20T02:51:00.429 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:00.430 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:00.428+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:00.430 INFO:teuthology.orchestra.run.smithi105.stderr:marked in osd.5. 2024-04-20T02:51:00.430 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-04-20T02:51:00.528 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:00.529+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:00.599 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:00.599 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:00.599 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:00.599 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:00.609 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:00.652 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:00.653+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:00.729 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:00.729+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:00.736 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:00.736 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:00.736 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:00.736 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:00.736 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:00.747 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:00.830 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:00.831+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:00.916 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:00.916 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:00.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:00.916 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.311804907999999, 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:00.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:51:00.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:51:00.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:51:00.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:51:00.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:51:00.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:51:00.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:51:00.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:29 4.8 4:1c68e542:::smithi10548823-8:head [write 517649~526743,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.558930277000002, 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031979782999999998, 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.924 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 29 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356788+0000", 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357534+0000", 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074630400000000004 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357547+0000", 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2697000000000001e-05 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357556+0000", 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8040000000000007e-06 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357617+0000", 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1038999999999996e-05 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357754+0000", 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013660500000000001 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357853+0000", 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9327000000000006e-05 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388575+0000", 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030721678999999998 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388623+0000", 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878999999999997e-05 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388660+0000", 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7002000000000002e-05 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388713+0000", 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2608999999999997e-05 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388769+0000", 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5896999999999997e-05 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.929 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:30 4.8 4:1c68e542:::smithi10548823-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.55165272, 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033039351000000002, 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 30 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364066+0000", 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364312+0000", 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000246708 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364328+0000", 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5376999999999999e-05 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364339+0000", 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.092e-05 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375692+0000", 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011353103999999999 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375805+0000", 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000113414 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375970+0000", 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016508399999999999 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388615+0000", 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012645145 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391156+0000", 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025406410000000002 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396990+0000", 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0058340390000000001 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397042+0000", 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2454000000000003e-05 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397106+0000", 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3632000000000003e-05 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.935 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:31 4.8 4:1c68e542:::smithi10548823-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.551366024, 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032910542000000001, 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 31 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.936 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:00.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364357+0000", 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1160000000000001e-06 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364363+0000", 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1680000000000001e-06 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364367+0000", 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8750000000000002e-06 2024-04-20T02:51:00.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388814+0000", 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024447437999999998 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388914+0000", 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9557999999999998e-05 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389062+0000", 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014799600000000001 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391160+0000", 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020979760000000001 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396038+0000", 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048784689999999999 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397155+0000", 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001116286 2024-04-20T02:51:00.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397208+0000", 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3273000000000003e-05 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397264+0000", 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5962999999999999e-05 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:32 4.8 4:1c68e542:::smithi10548823-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.551347310000001, 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033121840999999999, 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 32 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364375+0000", 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1149999999999998e-06 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364382+0000", 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4259999999999997e-06 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364387+0000", 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6129999999999997e-06 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389813+0000", 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025426621 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389905+0000", 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2011999999999999e-05 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.945 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397299+0000", 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007393897 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397395+0000", 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5293999999999999e-05 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397494+0000", 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9289000000000004e-05 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.946 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.547938509000002, 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.950 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.434635468, 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.954 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:66 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:00.956 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.429277580000001, 2024-04-20T02:51:00.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033365526, 2024-04-20T02:51:00.957 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.957 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 66 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.958 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486438+0000", 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486778+0000", 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000339709 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.959 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486798+0000", 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0462000000000002e-05 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486812+0000", 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3589e-05 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496889+0000", 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010077477 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496978+0000", 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8205e-05 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.497125+0000", 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000147665 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518444+0000", 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.021318713 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519407+0000", 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00096312999999999998 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519727+0000", 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031953500000000002 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519764+0000", 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7406000000000001e-05 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519807+0000", 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3402000000000002e-05 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:67 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2268912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.428892395999998, 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033307638000000001, 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.963 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 67 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:00.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486826+0000", 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486831+0000", 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0189999999999997e-06 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486839+0000", 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4710000000000001e-06 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486844+0000", 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.57e-06 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509240+0000", 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022395241 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509338+0000", 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8034000000000004e-05 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509498+0000", 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015993999999999999 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518471+0000", 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0089732730000000004 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519859+0000", 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013877130000000001 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520042+0000", 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182981 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520084+0000", 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2518999999999997e-05 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520135+0000", 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0491000000000002e-05 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.428869950999999, 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:00.969 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:51:00.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:51:00.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:161 4.3 e29/27 4:c32272c0:::smithi10548823-41:head v 29'16, mlcod=29'14)", 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.081433860000001, 2024-04-20T02:51:00.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031990785000000001, 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834285+0000", 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835168+0000", 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00088287800000000003 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835180+0000", 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2340999999999999e-05 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835187+0000", 2024-04-20T02:51:00.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7830000000000001e-06 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835227+0000", 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0754000000000003e-05 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835243+0000", 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5607000000000002e-05 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866234+0000", 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030990891 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866276+0000", 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2469999999999998e-05 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:00.978 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.074849816, 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:00.982 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.064486210999998, 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:00.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.986 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.056401732000001, 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.990 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:981 4.7 e32/27 4:e9876a63:::smithi10548823-30:head v 32'19, mlcod=30'17)", 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.326349258, 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033092985999999998, 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589369+0000", 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590134+0000", 2024-04-20T02:51:00.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076479 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590148+0000", 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3679999999999999e-05 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590157+0000", 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590209+0000", 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2237999999999997e-05 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590235+0000", 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6251e-05 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622430+0000", 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032194953999999998 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622463+0000", 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2780000000000001e-05 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1230 4.5 e38/22 4:a94dc531:::smithi10548823-32:head v 38'11, mlcod=29'8)", 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 6.5032038600000002, 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033312785999999997, 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:00.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412515+0000", 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412827+0000", 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031200999999999999 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412835+0000", 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.61e-06 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412842+0000", 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6990000000000003e-06 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412921+0000", 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8380000000000005e-05 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412944+0000", 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3609e-05 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445795+0000", 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032850279000000003 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445828+0000", 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3673000000000001e-05 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.77484350700000004, 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:00.999 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:01.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:51:01.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:51:01.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:51:01.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:51:01.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.76970639699999999, 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:01.005 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:01.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:01.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:01.010 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.76943338500000003, 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:01.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:01.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:51:01.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:51:01.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:01.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:51:01.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:51:01.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:01.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.76940399999999998, 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:01.017 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:51:01.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:51:01.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:51:01.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:51:01.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:01.022 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:01.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:01.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:01.022 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:01.022 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:01.022 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:01.024 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:01.024+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:01.024 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:01.114 INFO:tasks.rados.rados.0.smithi105.stdout:840: done (0 left) 2024-04-20T02:51:01.114 INFO:tasks.rados.rados.0.smithi105.stdout:841: write (excl) oid 35 current snap is 9 2024-04-20T02:51:01.114 INFO:tasks.rados.rados.0.smithi105.stdout:841: seq_num 124 ranges {484068=792524,1906473=177595} 2024-04-20T02:51:01.122 INFO:tasks.rados.rados.0.smithi105.stdout:841: writing smithi10548823-35 from 484068 to 1276592 tid 1 2024-04-20T02:51:01.124 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:01.124+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:01.125 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:51:01.127 INFO:tasks.rados.rados.0.smithi105.stdout:841: writing smithi10548823-35 from 1906473 to 2084068 tid 2 2024-04-20T02:51:01.127 INFO:tasks.rados.rados.0.smithi105.stdout:842: write (excl) oid 9 current snap is 9 2024-04-20T02:51:01.127 INFO:tasks.rados.rados.0.smithi105.stdout:842: seq_num 125 ranges {500875=467773,1564955=535920} 2024-04-20T02:51:01.137 INFO:tasks.rados.rados.0.smithi105.stdout:842: writing smithi10548823-9 from 500875 to 968648 tid 1 2024-04-20T02:51:01.141 INFO:tasks.rados.rados.0.smithi105.stdout:842: writing smithi10548823-9 from 1564955 to 2100875 tid 2 2024-04-20T02:51:01.141 INFO:tasks.rados.rados.0.smithi105.stdout:843: rollback oid 31 current snap is 9 2024-04-20T02:51:01.141 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 31 to 5 2024-04-20T02:51:01.141 INFO:tasks.rados.rados.0.smithi105.stdout:844: set_chunk oid 45 target oid 2024-04-20T02:51:01.164 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:01.164 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:01.164 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:01.164 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:01.167 INFO:tasks.rados.rados.0.smithi105.stdout:844: set_chunk oid 45 offset: 1144832 length: 1084416 target oid 40368e0ee489535481a1c8a22f4ee0b9a9e42e0f05ba89463cd55475a51c8e8f offset: 0 snap -1 2024-04-20T02:51:01.167 INFO:tasks.rados.rados.0.smithi105.stdout:845: read oid 38 snap 7 2024-04-20T02:51:01.167 INFO:tasks.rados.rados.0.smithi105.stdout:845: expect deleted 2024-04-20T02:51:01.168 INFO:tasks.rados.rados.0.smithi105.stdout:846: set_chunk oid 13 target oid 2024-04-20T02:51:01.176 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:01.191 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:01.211 INFO:tasks.rados.rados.0.smithi105.stdout:846: set_chunk oid 13 offset: 441344 length: 2009600 target oid ea1112aea5e7be379214f0429a6e898cf4f4a05bdbb29b6cd15764b4776418b1 offset: 0 snap -1 2024-04-20T02:51:01.211 INFO:tasks.rados.rados.0.smithi105.stdout:847: read oid 49 snap -1 2024-04-20T02:51:01.211 INFO:tasks.rados.rados.0.smithi105.stdout:847: expect (ObjNum 93 snap 3 seq_num 93) 2024-04-20T02:51:01.211 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:01.212 INFO:tasks.rados.rados.0.smithi105.stdout:848: set_chunk oid 30 target oid 2024-04-20T02:51:01.232 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:01.251 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:01.259 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:01.260+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:01.269 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:01.289 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:01.307 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:01.326 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:01.326+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:01.328 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:01.329 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:01.329 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:01.329 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:01.329 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:01.329 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:01.329 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:01.329 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:01.330 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:01.345 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:01.345 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:01.356 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:01.427 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:01.427+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:01.525 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:01.525 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:01.527 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:01.528+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:01.535 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:01.628 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:01.628+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:01.665 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:01.665 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:01.676 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:01.760 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:01.760+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:01.829 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:01.830+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:01.844 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:01.844 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:01.855 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:01.940 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:01.940+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:02.024 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:02.024 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:02.030 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:02.030+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:02.034 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:02.132 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:02.132+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:02.207 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:02.208 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:02.218 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:02.233 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:02.233+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:02.333 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:02.334+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:02.347 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:02.347 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:02.358 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:02.442 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:02.442+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:02.525 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:02.525 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:02.535 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:02.536 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:02.535+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:02.635 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:02.636+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:02.704 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:02.705 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:02.714 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:02.737 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:02.737+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:02.837 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:02.837+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:02.882 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:02.882 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:02.891 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:848: set_chunk oid 30 offset: 2107392 length: 819200 target oid d1d6aad75cbd3ed266c23d408be09838bb0d8e0bfc130470b78031d3f7d99d1f offset: 0 snap -1 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:849: delete oid 40 current snap is 9 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:843: finishing rollback tid 0 to smithi10548823-31 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:841: finishing write tid 1 to smithi10548823-35 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:843: finishing rollback tid 1 to smithi10548823-31 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:843: finishing rollback tid 2 to smithi10548823-31 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 47 (ObjNum 89 snap 2 seq_num 89) dirty exists 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:841: finishing write tid 2 to smithi10548823-35 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:841: finishing write tid 3 to smithi10548823-35 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:841: finishing write tid 4 to smithi10548823-35 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:841: oid 35 updating version 0 to 28 2024-04-20T02:51:02.953 INFO:tasks.rados.rados.0.smithi105.stdout:841: oid 35 version 28 is already newer than 27 2024-04-20T02:51:02.954 INFO:tasks.rados.rados.0.smithi105.stdout:841: oid 35 updating version 28 to 29 2024-04-20T02:51:02.954 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 29 (ObjNum 124 snap 9 seq_num 124) dirty exists 2024-04-20T02:51:02.954 INFO:tasks.rados.rados.0.smithi105.stdout:841: left oid 35 (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:51:02.954 INFO:tasks.rados.rados.0.smithi105.stdout:844: finishing set_chunk to oid 45 2024-04-20T02:51:02.954 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 1144832 length: 1084416 2024-04-20T02:51:02.954 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:51:02.954 INFO:tasks.rados.rados.0.smithi105.stdout:846: finishing set_chunk to oid 13 2024-04-20T02:51:02.954 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 61 (ObjNum 97 snap 4 seq_num 97) dirty exists 2024-04-20T02:51:02.954 INFO:tasks.rados.rados.0.smithi105.stdout:847: expect (ObjNum 93 snap 3 seq_num 93) 2024-04-20T02:51:02.955 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:02.952+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:03.039 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:03.040+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:03.071 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:03.071 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:03.083 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:03.114 INFO:tasks.rados.rados.0.smithi105.stdout:841: done (8 left) 2024-04-20T02:51:03.114 INFO:tasks.rados.rados.0.smithi105.stdout:843: done (7 left) 2024-04-20T02:51:03.114 INFO:tasks.rados.rados.0.smithi105.stdout:844: done (6 left) 2024-04-20T02:51:03.114 INFO:tasks.rados.rados.0.smithi105.stdout:845: done (5 left) 2024-04-20T02:51:03.114 INFO:tasks.rados.rados.0.smithi105.stdout:846: done (4 left) 2024-04-20T02:51:03.114 INFO:tasks.rados.rados.0.smithi105.stdout:847: done (3 left) 2024-04-20T02:51:03.114 INFO:tasks.rados.rados.0.smithi105.stdout:849: done (2 left) 2024-04-20T02:51:03.114 INFO:tasks.rados.rados.0.smithi105.stdout:850: write (excl) oid 39 current snap is 9 2024-04-20T02:51:03.114 INFO:tasks.rados.rados.0.smithi105.stdout:842: finishing write tid 1 to smithi10548823-9 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:842: finishing write tid 2 to smithi10548823-9 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:842: finishing write tid 3 to smithi10548823-9 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:842: finishing write tid 4 to smithi10548823-9 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:842: oid 9 updating version 0 to 100 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:842: oid 9 version 100 is already newer than 98 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:842: oid 9 version 100 is already newer than 99 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 100 (ObjNum 125 snap 9 seq_num 125) dirty exists 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:842: left oid 9 (ObjNum 125 snap 9 seq_num 125) 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:848: finishing set_chunk to oid 30 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 27 (ObjNum 107 snap 5 seq_num 107) dirty exists 2024-04-20T02:51:03.115 INFO:tasks.rados.rados.0.smithi105.stdout:850: seq_num 127 ranges {534489=618271,1565566=568923} 2024-04-20T02:51:03.120 INFO:tasks.rados.rados.0.smithi105.stdout:850: writing smithi10548823-39 from 534489 to 1152760 tid 1 2024-04-20T02:51:03.125 INFO:tasks.rados.rados.0.smithi105.stdout:850: writing smithi10548823-39 from 1565566 to 2134489 tid 2 2024-04-20T02:51:03.125 INFO:tasks.rados.rados.0.smithi105.stdout:842: done (2 left) 2024-04-20T02:51:03.126 INFO:tasks.rados.rados.0.smithi105.stdout:848: done (1 left) 2024-04-20T02:51:03.126 INFO:tasks.rados.rados.0.smithi105.stdout:851: write (excl) oid 34 current snap is 9 2024-04-20T02:51:03.126 INFO:tasks.rados.rados.0.smithi105.stdout:851: seq_num 128 ranges {551296=693520,2024048=127248} 2024-04-20T02:51:03.132 INFO:tasks.rados.rados.0.smithi105.stdout:851: writing smithi10548823-34 from 551296 to 1244816 tid 1 2024-04-20T02:51:03.138 INFO:tasks.rados.rados.0.smithi105.stdout:851: writing smithi10548823-34 from 2024048 to 2151296 tid 2 2024-04-20T02:51:03.138 INFO:tasks.rados.rados.0.smithi105.stdout:852: write (excl) oid 33 current snap is 9 2024-04-20T02:51:03.138 INFO:tasks.rados.rados.0.smithi105.stdout:852: seq_num 129 ranges {568103=768769,2082530=85573} 2024-04-20T02:51:03.139 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:03.140+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:03.142 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:51:03.147 INFO:tasks.rados.rados.0.smithi105.stdout:852: writing smithi10548823-33 from 568103 to 1336872 tid 1 2024-04-20T02:51:03.152 INFO:tasks.rados.rados.0.smithi105.stdout:852: writing smithi10548823-33 from 2082530 to 2168103 tid 2 2024-04-20T02:51:03.152 INFO:tasks.rados.rados.0.smithi105.stdout:853: read oid 31 snap -1 2024-04-20T02:51:03.152 INFO:tasks.rados.rados.0.smithi105.stdout:853: expect (ObjNum 89 snap 2 seq_num 89) 2024-04-20T02:51:03.152 INFO:tasks.rados.rados.0.smithi105.stdout:854: write oid 23 current snap is 9 2024-04-20T02:51:03.152 INFO:tasks.rados.rados.0.smithi105.stdout:854: seq_num 130 ranges {584910=560371,1857365=327545} 2024-04-20T02:51:03.159 INFO:tasks.rados.rados.0.smithi105.stdout:854: writing smithi10548823-23 from 584910 to 1145281 tid 1 2024-04-20T02:51:03.175 INFO:tasks.rados.rados.0.smithi105.stdout:854: writing smithi10548823-23 from 1857365 to 2184910 tid 2 2024-04-20T02:51:03.175 INFO:tasks.rados.rados.0.smithi105.stdout:855: tier_evict oid 29 2024-04-20T02:51:03.175 INFO:tasks.rados.rados.0.smithi105.stdout:855: tier_evict oid 29 snap -1 2024-04-20T02:51:03.175 INFO:tasks.rados.rados.0.smithi105.stdout:856: write oid 49 current snap is 9 2024-04-20T02:51:03.175 INFO:tasks.rados.rados.0.smithi105.stdout:856: seq_num 131 ranges {601717=635620,1799494=402223} 2024-04-20T02:51:03.175 INFO:tasks.rados.rados.0.smithi105.stdout:856: writing smithi10548823-49 from 601717 to 1237337 tid 1 2024-04-20T02:51:03.180 INFO:tasks.rados.rados.0.smithi105.stdout:856: writing smithi10548823-49 from 1799494 to 2201717 tid 2 2024-04-20T02:51:03.191 INFO:tasks.rados.rados.0.smithi105.stdout:857: rollback oid 15 current snap is 9 2024-04-20T02:51:03.191 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 15 to 7 2024-04-20T02:51:03.192 INFO:tasks.rados.rados.0.smithi105.stdout:858: read oid 40 snap -1 2024-04-20T02:51:03.192 INFO:tasks.rados.rados.0.smithi105.stdout:858: expect deleted 2024-04-20T02:51:03.192 INFO:tasks.rados.rados.0.smithi105.stdout:859: tier_flush oid 17 2024-04-20T02:51:03.192 INFO:tasks.rados.rados.0.smithi105.stdout:859: tier_flush oid 17 snap -1 2024-04-20T02:51:03.192 INFO:tasks.rados.rados.0.smithi105.stdout:860: write (excl) oid 24 current snap is 9 2024-04-20T02:51:03.192 INFO:tasks.rados.rados.0.smithi105.stdout:860: seq_num 132 ranges {618524=710869,1857976=360548} 2024-04-20T02:51:03.192 INFO:tasks.rados.rados.0.smithi105.stdout:860: writing smithi10548823-24 from 618524 to 1329393 tid 1 2024-04-20T02:51:03.205 INFO:tasks.rados.rados.0.smithi105.stdout:860: writing smithi10548823-24 from 1857976 to 2218524 tid 2 2024-04-20T02:51:03.206 INFO:tasks.rados.rados.0.smithi105.stdout:861: tier_flush oid 21 2024-04-20T02:51:03.206 INFO:tasks.rados.rados.0.smithi105.stdout:861: tier_flush oid 21 snap -1 2024-04-20T02:51:03.206 INFO:tasks.rados.rados.0.smithi105.stdout:862: write (excl) oid 1 current snap is 9 2024-04-20T02:51:03.206 INFO:tasks.rados.rados.0.smithi105.stdout:862: seq_num 133 ranges {635331=786118,1916458=318873} 2024-04-20T02:51:03.206 INFO:tasks.rados.rados.0.smithi105.stdout:862: writing smithi10548823-1 from 635331 to 1421449 tid 1 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout:862: writing smithi10548823-1 from 1916458 to 2235331 tid 2 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout:863: tier_promote oid 30 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout:864: rollback oid 41 current snap is 9 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 41 to 6 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout:865: read oid 50 snap -1 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout:865: expect deleted 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout:850: finishing write tid 1 to smithi10548823-39 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout:851: finishing write tid 1 to smithi10548823-34 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout:851: finishing write tid 2 to smithi10548823-34 2024-04-20T02:51:03.210 INFO:tasks.rados.rados.0.smithi105.stdout:851: finishing write tid 3 to smithi10548823-34 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:851: finishing write tid 4 to smithi10548823-34 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:851: oid 34 updating version 0 to 103 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:851: oid 34 version 103 is already newer than 102 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:851: oid 34 updating version 103 to 104 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 104 (ObjNum 128 snap 9 seq_num 128) dirty exists 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:851: left oid 34 (ObjNum 128 snap 9 seq_num 128) 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:850: finishing write tid 2 to smithi10548823-39 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:850: finishing write tid 3 to smithi10548823-39 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:850: finishing write tid 4 to smithi10548823-39 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:850: oid 39 updating version 0 to 53 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:850: oid 39 updating version 53 to 54 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:850: oid 39 version 54 is already newer than 52 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 54 (ObjNum 127 snap 9 seq_num 127) dirty exists 2024-04-20T02:51:03.211 INFO:tasks.rados.rados.0.smithi105.stdout:850: left oid 39 (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:03.212 INFO:tasks.rados.rados.0.smithi105.stdout:855: got (22) Invalid argument 2024-04-20T02:51:03.212 INFO:tasks.rados.rados.0.smithi105.stdout:853: expect (ObjNum 89 snap 2 seq_num 89) 2024-04-20T02:51:03.240 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:03.240+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:03.302 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:03.302 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:03.313 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:03.340 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:03.341 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:03.342+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:03.380 INFO:tasks.rados.rados.0.smithi105.stdout:857: finishing rollback tid 1 to smithi10548823-15 2024-04-20T02:51:03.380 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 0 (ObjNum 33 snap 0 seq_num 51355856) dirty dne 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:859: got (16) Device or resource busy 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:check_oldest_snap_flushed oid 17 v 46 (ObjNum 123 snap 7 seq_num 123) unflushed 7 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:850: done (15 left) 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:851: done (14 left) 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:853: done (13 left) 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:855: done (12 left) 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:857: done (11 left) 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:859: done (10 left) 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:866: tier_flush oid 20 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:866: tier_flush oid 20 snap -1 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:867: read oid 25 snap -1 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:867: expect (ObjNum 70 snap 0 seq_num 70) 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:868: write oid 2 current snap is 9 2024-04-20T02:51:03.381 INFO:tasks.rados.rados.0.smithi105.stdout:868: seq_num 134 ranges {652138=461367,1574940=677198} 2024-04-20T02:51:03.386 INFO:tasks.rados.rados.0.smithi105.stdout:868: writing smithi10548823-2 from 652138 to 1113505 tid 1 2024-04-20T02:51:03.394 INFO:tasks.rados.rados.0.smithi105.stdout:868: writing smithi10548823-2 from 1574940 to 2252138 tid 2 2024-04-20T02:51:03.394 INFO:tasks.rados.rados.0.smithi105.stdout:869: set_chunk oid 14 target oid 2024-04-20T02:51:03.442 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:03.443+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:03.443 INFO:tasks.rados.rados.0.smithi105.stdout:869: set_chunk oid 14 offset: 15872 length: 524288 target oid 755c26938c0298d690d4d0806ec3cf90e5f2e39e0b3668d5788dbd58d703f1a8 offset: 0 snap -1 2024-04-20T02:51:03.443 INFO:tasks.rados.rados.0.smithi105.stdout:870: set_chunk oid 17 target oid 2024-04-20T02:51:03.530 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:03.531 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:03.541 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:03.543 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:03.543+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:03.544 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:03.564 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:03.585 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:03.604 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:03.624 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:03.644 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:03.644+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:03.644 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:03.664 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:03.679 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:03.679 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:03.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:03.679 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:03.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.679 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:941 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'19, mlcod=30'17)", 2024-04-20T02:51:03.679 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:51:03.679 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.159966027999999, 2024-04-20T02:51:03.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042482077, 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518291+0000", 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518290+0000", 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.518988+0000", 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069878099999999999 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519003+0000", 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4756999999999999e-05 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519013+0000", 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4250000000000004e-06 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519071+0000", 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8628000000000002e-05 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.519106+0000", 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4817000000000002e-05 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560724+0000", 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041618334 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.560773+0000", 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8937999999999998e-05 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:942 4.c e32/22 4:392c1e2f:::smithi10548823-44:head v 32'20, mlcod=30'17)", 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:51:03.683 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.141148918999999, 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030979784999999999, 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537108+0000", 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.537107+0000", 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538354+0000", 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001247232 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538381+0000", 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7008999999999998e-05 2024-04-20T02:51:03.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538395+0000", 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4093999999999999e-05 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538476+0000", 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1045999999999996e-05 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.538505+0000", 2024-04-20T02:51:03.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8969999999999999e-05 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568053+0000", 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029547370999999999 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.568088+0000", 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5540000000000002e-05 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.687 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.134517692999999, 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:51:03.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:51:03.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:51:03.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:51:03.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.128296619, 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.693 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:03.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:51:03.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:51:03.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:51:03.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:51:03.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:51:03.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:51:03.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:51:03.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:51:03.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:51:03.702 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.702 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.119896414999999, 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:51:03.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:51:03.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:03.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.708 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:971 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (56) in=7b,truncate 3277332] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.119578346000001, 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051333776999999997, 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 971 2024-04-20T02:51:03.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558678+0000", 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558686+0000", 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7330000000000003e-06 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558693+0000", 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.866e-06 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558698+0000", 2024-04-20T02:51:03.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.49e-06 2024-04-20T02:51:03.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594685+0000", 2024-04-20T02:51:03.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035986753000000003 2024-04-20T02:51:03.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594792+0000", 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107129 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594958+0000", 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000165429 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595721+0000", 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076360100000000004 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.596242+0000", 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052083800000000003 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609915+0000", 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013673431 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609958+0000", 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2759000000000002e-05 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610013+0000", 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4421e-05 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.11955377, 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:03.715 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:03.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:03.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:51:03.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:51:03.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:51:03.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:51:03.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:51:03.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:51:03.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop_reply(client.4431.0:968 4.0 e32/22 ondisk, result = 0)", 2024-04-20T02:51:03.721 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:51:03.722 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.112113484999998, 2024-04-20T02:51:03.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028443566999999999, 2024-04-20T02:51:03.722 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.722 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:03.722 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:51:03.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566144+0000", 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566143+0000", 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566152+0000", 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1759999999999999e-06 2024-04-20T02:51:03.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566163+0000", 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0852999999999999e-05 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.566172+0000", 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2229999999999996e-06 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594564+0000", 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028392473000000001 2024-04-20T02:51:03.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594574+0000", 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5899999999999997e-06 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594587+0000", 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3454e-05 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.725 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:986 4.f e32/22 4:f72cd604:::smithi10548823-29:head v 32'5, mlcod=29'3)", 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.042052728000002, 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.040291053, 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636204+0000", 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636467+0000", 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026249800000000001 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636478+0000", 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1762e-05 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636493+0000", 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4826e-05 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636546+0000", 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2638e-05 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636570+0000", 2024-04-20T02:51:03.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4235000000000001e-05 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676457+0000", 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039887226999999997 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676496+0000", 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8281000000000002e-05 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1002 4.0 4:0158e114:::smithi10548823-21:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:03.729 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.935818913999999, 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042474603999999999, 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1002 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742438+0000", 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742436+0000", 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742449+0000", 2024-04-20T02:51:03.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3042e-05 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742469+0000", 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0007999999999999e-05 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742481+0000", 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1297999999999999e-05 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742528+0000", 2024-04-20T02:51:03.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7697000000000001e-05 2024-04-20T02:51:03.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.742598+0000", 2024-04-20T02:51:03.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9638e-05 2024-04-20T02:51:03.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.784913+0000", 2024-04-20T02:51:03.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042314928000000002 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1166 4.9 e35/22 4:99bb9a71:::smithi10548823-15:head v 35'51, mlcod=35'49)", 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:51.353617+0000", 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.324639748999999, 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026675606000000001, 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.353617+0000", 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.353617+0000", 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.353616+0000", 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354591+0000", 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00097470900000000004 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354609+0000", 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7671000000000001e-05 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354618+0000", 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0420000000000008e-06 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354665+0000", 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7265000000000002e-05 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.354689+0000", 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3659000000000002e-05 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.380236+0000", 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025547367000000001 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.380293+0000", 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7093999999999999e-05 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.3132317669999996, 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.738 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:51:03.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:51:03.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.3059688049999991, 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.744 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:51:03.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:51:03.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.3000262760000005, 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:51:03.750 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.2998232840000004, 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.756 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.761 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.2998003570000005, 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.766 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.2897166230000003, 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:03.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:51:03.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:03.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:51:03.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:51:03.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.2728312150000001, 2024-04-20T02:51:03.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:03.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:51:03.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.775 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.2565547739999996, 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:51:03.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:51:03.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:51:03.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1397 4.9 e42/22 4:95e19a88:::smithi10548823-40:head v 42'88, mlcod=42'86)", 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:59.023037+0000", 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.6552198819999999, 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038292596999999998, 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:03.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.023037+0000", 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.023037+0000", 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.023036+0000", 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.024064+0000", 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010281069999999999 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.024099+0000", 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4712000000000002e-05 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.024110+0000", 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0927999999999999e-05 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:03.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.024157+0000", 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7162e-05 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.024177+0000", 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9942e-05 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.061292+0000", 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037114644000000002 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:59.061330+0000", 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8282999999999999e-05 2024-04-20T02:51:03.783 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.784 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.784 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.784 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:03.784 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:03.784 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:03.785 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:03.785 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:03.785 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:03.785 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:03.785 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:03.785 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:03.785 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:03.785 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:03.786 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:03.787 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:03.788 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:03.786+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:03.886 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:03.886+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:03.916 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:03.916 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:03.916 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:03.916 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:03.927 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:04.011 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:04.011+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:04.088 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:04.088+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:04.095 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:04.095 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:04.095 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:04.095 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:04.095 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:04.105 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:04.189 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:04.189+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:04.277 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:04.277 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.333422596, 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:51:04.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:51:04.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:51:04.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:51:04.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:51:04.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.325944278, 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.284 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:51:04.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:51:04.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1146 4.d 4:b376c20e:::smithi10548823-49:head [write 3437750~125301] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.321376417, 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070707487999999999, 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.290 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1146 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955022+0000", 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955179+0000", 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015657 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955188+0000", 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.645e-06 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955195+0000", 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2910000000000004e-06 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978329+0000", 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02313401 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978450+0000", 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121438 2024-04-20T02:51:04.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978652+0000", 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020163700000000001 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984380+0000", 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057282510000000002 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991325+0000", 2024-04-20T02:51:04.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069446830000000001 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025680+0000", 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034355284 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025709+0000", 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8781000000000001e-05 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025731+0000", 2024-04-20T02:51:04.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1963e-05 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1147 4.d 4:b376c20e:::smithi10548823-49:head [setxattr _header (56) in=7b,truncate 3563051] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.32119808, 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070713812000000001, 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1147 2024-04-20T02:51:04.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955201+0000", 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955204+0000", 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.05e-06 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955207+0000", 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1949999999999999e-06 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955210+0000", 2024-04-20T02:51:04.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983351+0000", 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028140978000000001 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983488+0000", 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013647 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:04.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983668+0000", 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000180335 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984487+0000", 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081847099999999998 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991393+0000", 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069068569999999998 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025852+0000", 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034458946999999997 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025883+0000", 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0261e-05 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025915+0000", 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2654999999999998e-05 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.301 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.321186379, 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:51:04.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:04.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:51:04.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:51:04.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8977574449999999, 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8977205619999999, 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.313 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.314 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.314 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:51:04.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.314 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:04.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:04.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:04.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:51:04.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:04.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.323 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8976959289999993, 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.324 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:51:04.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.328 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8912025180000001, 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.329 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:51:04.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:51:04.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:04.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.335 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8851384959999997, 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:04.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:04.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8750767310000001, 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.342 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:51:04.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:04.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8743841270000008, 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.348 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:04.349 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:51:04.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:51:04.353 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8743724729999993, 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.354 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:51:04.358 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.1683079169999999, 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:51:04.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:51:04.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:51:04.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:51:04.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:51:04.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:51:04.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:51:04.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1426 4.1 4:8b927dd6:::smithi10548823-28:head [write 1331027~582033] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.1567537550000004, 2024-04-20T02:51:04.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068185170000000003, 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1426 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119645+0000", 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120235+0000", 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059073600000000002 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120249+0000", 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4168000000000001e-05 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120258+0000", 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.2719999999999997e-06 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124047+0000", 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003788839 2024-04-20T02:51:04.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124154+0000", 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000107747 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124307+0000", 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000152998 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148497+0000", 2024-04-20T02:51:04.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024189789999999999 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168007+0000", 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019509463000000001 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187765+0000", 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019758738000000001 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187802+0000", 2024-04-20T02:51:04.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7153999999999997e-05 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187831+0000", 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8592999999999999e-05 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1428 4.1 4:8b927dd6:::smithi10548823-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.1561009259999997, 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067836666000000004, 2024-04-20T02:51:04.369 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1428 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120298+0000", 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120301+0000", 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.853e-06 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120308+0000", 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0559999999999999e-06 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120313+0000", 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1680000000000003e-06 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148583+0000", 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028269882 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148673+0000", 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9922999999999999e-05 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187934+0000", 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039260647000000003 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188031+0000", 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7468999999999998e-05 2024-04-20T02:51:04.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188135+0000", 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104126 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.1492472009999997, 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:51:04.374 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:51:04.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:51:04.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.1433008869999997, 2024-04-20T02:51:04.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:51:04.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:51:04.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:51:04.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.1427354459999997, 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:51:04.386 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:51:04.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:51:04.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.1427094149999997, 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:51:04.392 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:51:04.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:51:04.397 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.397 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.397 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.397 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.397 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.397 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:04.399 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:04.399 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:04.398+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:04.416 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:51:04.499 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:04.500+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:04.530 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:04.531 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:04.531 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:04.531 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:04.541 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:04.600 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:04.600+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:04.701 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:04.702+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:04.709 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:04.709 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:04.709 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:04.709 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:04.709 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:04.722 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:04.802 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:04.802+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:04.853 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:04.853 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.117737404, 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.854 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:51:04.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:51:04.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.086448343000001, 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.860 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:04.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:51:04.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:51:04.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:51:04.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:51:04.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:51:04.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:51:04.866 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.866 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.866 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.081757886999998, 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:51:04.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:51:04.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:04.873 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.075001417999999, 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.879 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.065185907, 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:04.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:04.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:51:04.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:51:04.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:51:04.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:04.886 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.064141974999998, 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.892 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.064120814999999, 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:04.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:51:04.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.856550234, 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:51:04.897 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:04.901 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.842355589, 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.836388331999999, 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.906 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:51:04.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:51:04.910 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.827982329000001, 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:51:04.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.724350288, 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.915 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.919 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.493753978000001, 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:04.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:51:04.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.466809828999999, 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:04.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:51:04.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.930 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.441451562000001, 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:04.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:51:04.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.413311117999999, 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:51:04.935 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.936 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:51:04.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:51:04.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.375942512999998, 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:51:04.943 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.375645590000001, 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:51:04.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:51:04.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:04.948 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.370562472, 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:51:04.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:51:04.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.343559913, 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:51:04.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:51:04.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:04.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:51:04.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:51:04.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.957 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:04.957 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:04.957 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:04.959 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:04.959 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:04.959+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:05.059 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:05.060+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:05.130 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:05.131 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:05.131 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:05.131 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:05.138 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:05.178 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:05.179+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:05.261 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:05.261+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:05.305 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:05.305 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:05.306 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:05.306 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:05.306 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:05.316 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:05.362 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:05.362+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:05.431 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:51:05.431 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:870: set_chunk oid 17 offset: 1168896 length: 374784 target oid 49dfa3a36ea566ece109be00940c25f90e74796f7ebe644247ba8ecd48263ae5 offset: 0 snap 8 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:871: read oid 32 snap 8 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:871: expect (ObjNum 113 snap 7 seq_num 113) 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:852: finishing write tid 1 to smithi10548823-33 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:861: got (0) Success 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:852: finishing write tid 2 to smithi10548823-33 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:852: finishing write tid 3 to smithi10548823-33 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:858: done (15 left) 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:861: done (14 left) 2024-04-20T02:51:05.435 INFO:tasks.rados.rados.0.smithi105.stdout:872: read oid 40 snap 5 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:872: expect (ObjNum 100 snap 5 seq_num 100) 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:873: read oid 4 snap -1 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:873: expect (ObjNum 96 snap 4 seq_num 96) 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:852: finishing write tid 4 to smithi10548823-33 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:852: oid 33 updating version 0 to 23 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:852: oid 33 updating version 23 to 24 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:852: oid 33 updating version 24 to 25 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 25 (ObjNum 129 snap 9 seq_num 129) dirty exists 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:852: left oid 33 (ObjNum 129 snap 9 seq_num 129) 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:854: finishing write tid 1 to smithi10548823-23 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:854: finishing write tid 2 to smithi10548823-23 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:852: done (15 left) 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:865: done (14 left) 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:874: read oid 50 snap -1 2024-04-20T02:51:05.436 INFO:tasks.rados.rados.0.smithi105.stdout:874: expect deleted 2024-04-20T02:51:05.437 INFO:tasks.rados.rados.0.smithi105.stdout:875: write (excl) oid 44 current snap is 9 2024-04-20T02:51:05.437 INFO:tasks.rados.rados.0.smithi105.stdout:875: seq_num 135 ranges {668945=536616,1633422=491363} 2024-04-20T02:51:05.441 INFO:tasks.rados.rados.0.smithi105.stdout:875: writing smithi10548823-44 from 668945 to 1205561 tid 1 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:875: writing smithi10548823-44 from 1633422 to 2124785 tid 2 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:854: finishing write tid 3 to smithi10548823-23 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:854: finishing write tid 4 to smithi10548823-23 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:854: oid 23 updating version 0 to 35 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:854: oid 23 updating version 35 to 36 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:854: oid 23 updating version 36 to 37 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 37 (ObjNum 130 snap 9 seq_num 130) dirty exists 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:854: left oid 23 (ObjNum 130 snap 9 seq_num 130) 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:856: finishing write tid 1 to smithi10548823-49 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:856: finishing write tid 2 to smithi10548823-49 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:854: done (15 left) 2024-04-20T02:51:05.447 INFO:tasks.rados.rados.0.smithi105.stdout:876: set_chunk oid 26 target oid 2024-04-20T02:51:05.463 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:05.463+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:05.490 INFO:tasks.rados.rados.0.smithi105.stdout:876: set_chunk oid 26 offset: 911872 length: 1404416 target oid 6bf6bf16857d71fb38584e76251c18e31115f598247b2f307e76d4cd6c33476c offset: 0 snap -1 2024-04-20T02:51:05.490 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:05.490 INFO:tasks.rados.rados.0.smithi105.stdout:856: finishing write tid 3 to smithi10548823-49 2024-04-20T02:51:05.490 INFO:tasks.rados.rados.0.smithi105.stdout:856: finishing write tid 4 to smithi10548823-49 2024-04-20T02:51:05.490 INFO:tasks.rados.rados.0.smithi105.stdout:856: oid 49 updating version 0 to 50 2024-04-20T02:51:05.490 INFO:tasks.rados.rados.0.smithi105.stdout:856: oid 49 version 50 is already newer than 49 2024-04-20T02:51:05.490 INFO:tasks.rados.rados.0.smithi105.stdout:856: oid 49 updating version 50 to 51 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 51 (ObjNum 131 snap 9 seq_num 131) dirty exists 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:856: left oid 49 (ObjNum 131 snap 9 seq_num 131) 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:860: finishing write tid 1 to smithi10548823-24 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:856: done (15 left) 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:877: rollback oid 7 current snap is 9 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 7 to 6 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:860: finishing write tid 2 to smithi10548823-24 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:860: finishing write tid 3 to smithi10548823-24 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:860: finishing write tid 4 to smithi10548823-24 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:860: oid 24 updating version 0 to 96 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:860: oid 24 updating version 96 to 98 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:860: oid 24 version 98 is already newer than 97 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 98 (ObjNum 132 snap 9 seq_num 132) dirty exists 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:860: left oid 24 (ObjNum 132 snap 9 seq_num 132) 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:862: finishing write tid 1 to smithi10548823-1 2024-04-20T02:51:05.491 INFO:tasks.rados.rados.0.smithi105.stdout:862: finishing write tid 2 to smithi10548823-1 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:862: finishing write tid 3 to smithi10548823-1 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:862: finishing write tid 4 to smithi10548823-1 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:862: oid 1 updating version 0 to 107 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:862: oid 1 version 107 is already newer than 106 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:862: oid 1 updating version 107 to 108 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 108 (ObjNum 133 snap 9 seq_num 133) dirty exists 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:862: left oid 1 (ObjNum 133 snap 9 seq_num 133) 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:863: got (0) Success 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 27 (ObjNum 107 snap 5 seq_num 107) dirty exists 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:864: finishing rollback tid 0 to smithi10548823-41 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:860: done (15 left) 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:862: done (14 left) 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:863: done (13 left) 2024-04-20T02:51:05.492 INFO:tasks.rados.rados.0.smithi105.stdout:878: snap_remove snap 7 2024-04-20T02:51:05.505 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:05.505 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:05.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:05.505 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:05.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.900618793, 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:05.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:51:05.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:29 4.8 4:1c68e542:::smithi10548823-8:head [write 517649~526743,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:05.512 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.147744161999999, 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031979782999999998, 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 29 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:51:05.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356788+0000", 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357534+0000", 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074630400000000004 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357547+0000", 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2697000000000001e-05 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357556+0000", 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8040000000000007e-06 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357617+0000", 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1038999999999996e-05 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357754+0000", 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013660500000000001 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357853+0000", 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9327000000000006e-05 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388575+0000", 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030721678999999998 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388623+0000", 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878999999999997e-05 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388660+0000", 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7002000000000002e-05 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388713+0000", 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2608999999999997e-05 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388769+0000", 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5896999999999997e-05 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.518 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:30 4.8 4:1c68e542:::smithi10548823-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.140466605, 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033039351000000002, 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 30 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364066+0000", 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364312+0000", 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000246708 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364328+0000", 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5376999999999999e-05 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364339+0000", 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.092e-05 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375692+0000", 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011353103999999999 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375805+0000", 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000113414 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375970+0000", 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016508399999999999 2024-04-20T02:51:05.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388615+0000", 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012645145 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391156+0000", 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025406410000000002 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396990+0000", 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0058340390000000001 2024-04-20T02:51:05.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397042+0000", 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2454000000000003e-05 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397106+0000", 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3632000000000003e-05 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:31 4.8 4:1c68e542:::smithi10548823-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.140179909, 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032910542000000001, 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 31 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364357+0000", 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1160000000000001e-06 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364363+0000", 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1680000000000001e-06 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364367+0000", 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8750000000000002e-06 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388814+0000", 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024447437999999998 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388914+0000", 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9557999999999998e-05 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389062+0000", 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014799600000000001 2024-04-20T02:51:05.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391160+0000", 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020979760000000001 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396038+0000", 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048784689999999999 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397155+0000", 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001116286 2024-04-20T02:51:05.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397208+0000", 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3273000000000003e-05 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397264+0000", 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5962999999999999e-05 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:32 4.8 4:1c68e542:::smithi10548823-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.140161195000001, 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033121840999999999, 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 32 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364375+0000", 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1149999999999998e-06 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364382+0000", 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4259999999999997e-06 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364387+0000", 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6129999999999997e-06 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389813+0000", 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025426621 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389905+0000", 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2011999999999999e-05 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397299+0000", 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007393897 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397395+0000", 2024-04-20T02:51:05.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5293999999999999e-05 2024-04-20T02:51:05.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397494+0000", 2024-04-20T02:51:05.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9289000000000004e-05 2024-04-20T02:51:05.535 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.136752393999998, 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.538 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:51:05.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:51:05.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.023449353, 2024-04-20T02:51:05.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:51:05.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:51:05.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:51:05.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:51:05.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:51:05.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:66 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.018091465000001, 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033365526, 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 66 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486438+0000", 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486778+0000", 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000339709 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486798+0000", 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0462000000000002e-05 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486812+0000", 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3589e-05 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496889+0000", 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010077477 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496978+0000", 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8205e-05 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.497125+0000", 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000147665 2024-04-20T02:51:05.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518444+0000", 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.021318713 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519407+0000", 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00096312999999999998 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519727+0000", 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031953500000000002 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519764+0000", 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7406000000000001e-05 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519807+0000", 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3402000000000002e-05 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.554 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:67 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2268912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.017706280999999, 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033307638000000001, 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 67 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486826+0000", 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486831+0000", 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0189999999999997e-06 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486839+0000", 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4710000000000001e-06 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486844+0000", 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.57e-06 2024-04-20T02:51:05.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509240+0000", 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022395241 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509338+0000", 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8034000000000004e-05 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509498+0000", 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015993999999999999 2024-04-20T02:51:05.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518471+0000", 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0089732730000000004 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519859+0000", 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013877130000000001 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520042+0000", 2024-04-20T02:51:05.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182981 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520084+0000", 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2518999999999997e-05 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520135+0000", 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0491000000000002e-05 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.017683836, 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:05.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:51:05.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:51:05.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:161 4.3 e29/27 4:c32272c0:::smithi10548823-41:head v 29'16, mlcod=29'14)", 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.670247745000001, 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031990785000000001, 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834285+0000", 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835168+0000", 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00088287800000000003 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835180+0000", 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2340999999999999e-05 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835187+0000", 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7830000000000001e-06 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835227+0000", 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0754000000000003e-05 2024-04-20T02:51:05.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835243+0000", 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5607000000000002e-05 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866234+0000", 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030990891 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866276+0000", 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2469999999999998e-05 2024-04-20T02:51:05.569 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.663663701000001, 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:51:05.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.653300095999999, 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:05.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:51:05.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:51:05.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.645215617000002, 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:51:05.578 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:981 4.7 e32/27 4:e9876a63:::smithi10548823-30:head v 32'19, mlcod=30'17)", 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.915163143000001, 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033092985999999998, 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589369+0000", 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590134+0000", 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076479 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590148+0000", 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3679999999999999e-05 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590157+0000", 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590209+0000", 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2237999999999997e-05 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590235+0000", 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6251e-05 2024-04-20T02:51:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622430+0000", 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032194953999999998 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622463+0000", 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2780000000000001e-05 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1230 4.5 e38/22 4:a94dc531:::smithi10548823-32:head v 38'11, mlcod=29'8)", 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 11.092017745, 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033312785999999997, 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412515+0000", 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412827+0000", 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031200999999999999 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412835+0000", 2024-04-20T02:51:05.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.61e-06 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412842+0000", 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6990000000000003e-06 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412921+0000", 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8380000000000005e-05 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412944+0000", 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3609e-05 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445795+0000", 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032850279000000003 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445828+0000", 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3673000000000001e-05 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.590 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.3636573920000004, 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.591 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:05.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:51:05.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:51:05.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:51:05.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.596 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.3585202819999997, 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:51:05.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:51:05.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:51:05.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:51:05.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.3582472699999997, 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:51:05.602 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.3582178850000002, 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:05.608 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:51:05.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:51:05.612 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.613 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:05.613 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:05.613 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:05.614 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:05.615 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:05.615+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:05.715 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:05.715+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:05.783 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:05.783 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:05.784 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:05.784 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:05.788 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:05.794 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:05.837 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:05.838+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:05.917 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:05.917+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:05.921 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:05.921 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:05.921 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:05.922 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:05.922 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:05.931 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:05.932 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:05.950 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:05.968 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:05.987 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:06.006 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:06.018 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:06.018+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:06.025 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:06.044 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:06.063 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:06.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:06.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:06.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:06.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:06.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:06.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:06.065 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:06.065 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:06.102 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:06.102 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:06.112 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:06.113 INFO:tasks.rados.rados.0.smithi105.stdout:878: done (13 left) 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout:879: read oid 38 snap -1 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout:879: expect deleted 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout:880: read oid 8 snap -1 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout:880: expect (ObjNum 86 snap 2 seq_num 86) 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout:881: tier_evict oid 1 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout:881: tier_evict oid 1 snap -1 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout:864: finishing rollback tid 1 to smithi10548823-41 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout:864: finishing rollback tid 2 to smithi10548823-41 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 112 (ObjNum 87 snap 2 seq_num 87) dirty exists 2024-04-20T02:51:06.114 INFO:tasks.rados.rados.0.smithi105.stdout:867: expect (ObjNum 70 snap 0 seq_num 70) 2024-04-20T02:51:06.125 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.0 to 1 (10000) 2024-04-20T02:51:06.125 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:06.125+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:06.141 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-20T02:51:06.226 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:06.226+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:06.252 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:06.253 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:06.259 INFO:tasks.rados.rados.0.smithi105.stdout:868: finishing write tid 1 to smithi10548823-2 2024-04-20T02:51:06.259 INFO:tasks.rados.rados.0.smithi105.stdout:868: finishing write tid 2 to smithi10548823-2 2024-04-20T02:51:06.259 INFO:tasks.rados.rados.0.smithi105.stdout:868: finishing write tid 3 to smithi10548823-2 2024-04-20T02:51:06.259 INFO:tasks.rados.rados.0.smithi105.stdout:868: finishing write tid 4 to smithi10548823-2 2024-04-20T02:51:06.259 INFO:tasks.rados.rados.0.smithi105.stdout:868: oid 2 updating version 0 to 55 2024-04-20T02:51:06.259 INFO:tasks.rados.rados.0.smithi105.stdout:868: oid 2 version 55 is already newer than 54 2024-04-20T02:51:06.259 INFO:tasks.rados.rados.0.smithi105.stdout:868: oid 2 version 55 is already newer than 53 2024-04-20T02:51:06.259 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 55 (ObjNum 134 snap 9 seq_num 134) dirty exists 2024-04-20T02:51:06.259 INFO:tasks.rados.rados.0.smithi105.stdout:868: left oid 2 (ObjNum 134 snap 9 seq_num 134) 2024-04-20T02:51:06.259 INFO:tasks.rados.rados.0.smithi105.stdout:869: finishing set_chunk to oid 14 2024-04-20T02:51:06.260 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 41 (ObjNum 99 snap 5 seq_num 99) dirty exists 2024-04-20T02:51:06.260 INFO:tasks.rados.rados.0.smithi105.stdout:864: done (15 left) 2024-04-20T02:51:06.260 INFO:tasks.rados.rados.0.smithi105.stdout:867: done (14 left) 2024-04-20T02:51:06.260 INFO:tasks.rados.rados.0.smithi105.stdout:868: done (13 left) 2024-04-20T02:51:06.260 INFO:tasks.rados.rados.0.smithi105.stdout:869: done (12 left) 2024-04-20T02:51:06.260 INFO:tasks.rados.rados.0.smithi105.stdout:882: delete oid 6 current snap is 9 2024-04-20T02:51:06.260 INFO:tasks.rados.rados.0.smithi105.stdout:873: expect (ObjNum 96 snap 4 seq_num 96) 2024-04-20T02:51:06.264 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:06.321 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:06.322+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:06.422 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:06.422+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:06.431 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:51:06.438 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:06.438 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:06.450 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:06.498 INFO:tasks.rados.rados.0.smithi105.stdout:871: expect (ObjNum 113 snap 7 seq_num 113) 2024-04-20T02:51:06.523 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:06.524+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:06.623 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:06.624 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:06.624 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:06.624+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:06.635 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:06.725 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:06.725+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:06.726 INFO:tasks.rados.rados.0.smithi105.stdout:866: got (0) Success 2024-04-20T02:51:06.726 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 0 (ObjNum 94 snap 3 seq_num 94) dirty exists 2024-04-20T02:51:06.726 INFO:tasks.rados.rados.0.smithi105.stdout:872: expect (ObjNum 100 snap 5 seq_num 100) 2024-04-20T02:51:06.805 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:06.805 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:06.815 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:06.826 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:06.826+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:06.894 INFO:tasks.rados.rados.0.smithi105.stdout:870: finishing set_chunk to oid 17 2024-04-20T02:51:06.894 INFO:tasks.rados.rados.0.smithi105.stdout:875: finishing write tid 1 to smithi10548823-44 2024-04-20T02:51:06.894 INFO:tasks.rados.rados.0.smithi105.stdout:875: finishing write tid 2 to smithi10548823-44 2024-04-20T02:51:06.894 INFO:tasks.rados.rados.0.smithi105.stdout:875: finishing write tid 3 to smithi10548823-44 2024-04-20T02:51:06.894 INFO:tasks.rados.rados.0.smithi105.stdout:875: finishing write tid 4 to smithi10548823-44 2024-04-20T02:51:06.894 INFO:tasks.rados.rados.0.smithi105.stdout:875: oid 44 updating version 0 to 36 2024-04-20T02:51:06.894 INFO:tasks.rados.rados.0.smithi105.stdout:875: oid 44 version 36 is already newer than 35 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:875: oid 44 updating version 36 to 37 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 37 (ObjNum 135 snap 9 seq_num 135) dirty exists 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:875: left oid 44 (ObjNum 135 snap 9 seq_num 135) 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:876: finishing set_chunk to oid 26 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 26 set_chunk returned error code -95 offset: 911872 length: 1404416 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:877: finishing rollback tid 0 to smithi10548823-7 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:877: finishing rollback tid 1 to smithi10548823-7 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:877: finishing rollback tid 2 to smithi10548823-7 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 116 (ObjNum 105 snap 5 seq_num 105) dirty exists 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:881: got (22) Invalid argument 2024-04-20T02:51:06.895 INFO:tasks.rados.rados.0.smithi105.stdout:880: expect (ObjNum 86 snap 2 seq_num 86) 2024-04-20T02:51:06.927 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:06.927+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:06.987 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:06.987 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:06.998 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:07.028 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:07.028+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:866: done (12 left) 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:870: done (11 left) 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:871: done (10 left) 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:872: done (9 left) 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:873: done (8 left) 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:874: done (7 left) 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:875: done (6 left) 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:876: done (5 left) 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:877: done (4 left) 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:879: done (3 left) 2024-04-20T02:51:07.044 INFO:tasks.rados.rados.0.smithi105.stdout:880: done (2 left) 2024-04-20T02:51:07.045 INFO:tasks.rados.rados.0.smithi105.stdout:881: done (1 left) 2024-04-20T02:51:07.045 INFO:tasks.rados.rados.0.smithi105.stdout:882: done (0 left) 2024-04-20T02:51:07.045 INFO:tasks.rados.rados.0.smithi105.stdout:883: read oid 49 snap -1 2024-04-20T02:51:07.045 INFO:tasks.rados.rados.0.smithi105.stdout:883: expect (ObjNum 131 snap 9 seq_num 131) 2024-04-20T02:51:07.045 INFO:tasks.rados.rados.0.smithi105.stdout:884: set_chunk oid 24 target oid 2024-04-20T02:51:07.061 INFO:tasks.rados.rados.0.smithi105.stdout:884: set_chunk oid 24 offset: 1492480 length: 801280 target oid c717caebab9ad64f16d41d2caf89481d085a32a4fde7166b99a301d86f17f64c offset: 0 snap 6 2024-04-20T02:51:07.061 INFO:tasks.rados.rados.0.smithi105.stdout:885: read oid 37 snap 6 2024-04-20T02:51:07.062 INFO:tasks.rados.rados.0.smithi105.stdout:885: expect (ObjNum 91 snap 2 seq_num 91) 2024-04-20T02:51:07.062 INFO:tasks.rados.rados.0.smithi105.stdout:886: read oid 20 snap -1 2024-04-20T02:51:07.062 INFO:tasks.rados.rados.0.smithi105.stdout:886: expect (ObjNum 94 snap 3 seq_num 94) 2024-04-20T02:51:07.062 INFO:tasks.rados.rados.0.smithi105.stdout:887: write oid 26 current snap is 9 2024-04-20T02:51:07.062 INFO:tasks.rados.rados.0.smithi105.stdout:887: seq_num 137 ranges {702559=403467,1750386=495032} 2024-04-20T02:51:07.067 INFO:tasks.rados.rados.0.smithi105.stdout:887: writing smithi10548823-26 from 702559 to 1106026 tid 1 2024-04-20T02:51:07.075 INFO:tasks.rados.rados.0.smithi105.stdout:887: writing smithi10548823-26 from 1750386 to 2245418 tid 2 2024-04-20T02:51:07.075 INFO:tasks.rados.rados.0.smithi105.stdout:888: read oid 11 snap -1 2024-04-20T02:51:07.075 INFO:tasks.rados.rados.0.smithi105.stdout:888: expect deleted 2024-04-20T02:51:07.075 INFO:tasks.rados.rados.0.smithi105.stdout:889: rollback oid 43 current snap is 9 2024-04-20T02:51:07.075 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 43 to 5 2024-04-20T02:51:07.075 INFO:tasks.rados.rados.0.smithi105.stdout:883: expect (ObjNum 131 snap 9 seq_num 131) 2024-04-20T02:51:07.119 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.1 to 1 (10000) 2024-04-20T02:51:07.120 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:51:07.129 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:07.130+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:07.136 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-20T02:51:07.186 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:07.186 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:07.197 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:07.203 INFO:tasks.rados.rados.0.smithi105.stdout:884: finishing set_chunk to oid 24 2024-04-20T02:51:07.204 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 1492480 length: 801280 2024-04-20T02:51:07.204 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 0 (ObjNum 132 snap 9 seq_num 132) dirty exists 2024-04-20T02:51:07.204 INFO:tasks.rados.rados.0.smithi105.stdout:886: expect (ObjNum 94 snap 3 seq_num 94) 2024-04-20T02:51:07.247 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:07.248+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:07.327 INFO:tasks.rados.rados.0.smithi105.stdout:889: finishing rollback tid 0 to smithi10548823-43 2024-04-20T02:51:07.328 INFO:tasks.rados.rados.0.smithi105.stdout:889: finishing rollback tid 1 to smithi10548823-43 2024-04-20T02:51:07.328 INFO:tasks.rados.rados.0.smithi105.stdout:889: finishing rollback tid 2 to smithi10548823-43 2024-04-20T02:51:07.328 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 59 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-20T02:51:07.328 INFO:tasks.rados.rados.0.smithi105.stdout:885: expect (ObjNum 91 snap 2 seq_num 91) 2024-04-20T02:51:07.331 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:07.332+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:07.336 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:07.336 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:07.347 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:07.432 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:07.433+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:07.517 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:07.517 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:07.527 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:07.534 INFO:tasks.rados.rados.0.smithi105.stdout:887: finishing write tid 1 to smithi10548823-26 2024-04-20T02:51:07.534 INFO:tasks.rados.rados.0.smithi105.stdout:887: finishing write tid 2 to smithi10548823-26 2024-04-20T02:51:07.534 INFO:tasks.rados.rados.0.smithi105.stdout:883: done (6 left) 2024-04-20T02:51:07.534 INFO:tasks.rados.rados.0.smithi105.stdout:884: done (5 left) 2024-04-20T02:51:07.534 INFO:tasks.rados.rados.0.smithi105.stdout:885: done (4 left) 2024-04-20T02:51:07.534 INFO:tasks.rados.rados.0.smithi105.stdout:886: done (3 left) 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:888: done (2 left) 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:889: done (1 left) 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:890: read oid 42 snap -1 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:890: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:891: read oid 45 snap 6 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:891: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:892: rollback oid 12 current snap is 9 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:887: finishing write tid 3 to smithi10548823-26 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:887: finishing write tid 4 to smithi10548823-26 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:887: oid 26 updating version 0 to 104 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:887: oid 26 updating version 104 to 106 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:887: oid 26 version 106 is already newer than 105 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 106 (ObjNum 137 snap 9 seq_num 137) dirty exists 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:887: left oid 26 (ObjNum 137 snap 9 seq_num 137) 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 12 to 6 2024-04-20T02:51:07.535 INFO:tasks.rados.rados.0.smithi105.stdout:887: done (3 left) 2024-04-20T02:51:07.536 INFO:tasks.rados.rados.0.smithi105.stdout:893: write oid 28 current snap is 9 2024-04-20T02:51:07.536 INFO:tasks.rados.rados.0.smithi105.stdout:893: seq_num 138 ranges {719366=478716,1808868=510498} 2024-04-20T02:51:07.536 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:07.534+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:07.540 INFO:tasks.rados.rados.0.smithi105.stdout:893: writing smithi10548823-28 from 719366 to 1198082 tid 1 2024-04-20T02:51:07.548 INFO:tasks.rados.rados.0.smithi105.stdout:893: writing smithi10548823-28 from 1808868 to 2319366 tid 2 2024-04-20T02:51:07.548 INFO:tasks.rados.rados.0.smithi105.stdout:894: read oid 41 snap -1 2024-04-20T02:51:07.548 INFO:tasks.rados.rados.0.smithi105.stdout:894: expect (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:51:07.548 INFO:tasks.rados.rados.0.smithi105.stdout:895: read oid 15 snap -1 2024-04-20T02:51:07.548 INFO:tasks.rados.rados.0.smithi105.stdout:895: expect deleted 2024-04-20T02:51:07.548 INFO:tasks.rados.rados.0.smithi105.stdout:896: read oid 16 snap 6 2024-04-20T02:51:07.549 INFO:tasks.rados.rados.0.smithi105.stdout:896: expect deleted 2024-04-20T02:51:07.549 INFO:tasks.rados.rados.0.smithi105.stdout:897: read oid 30 snap -1 2024-04-20T02:51:07.549 INFO:tasks.rados.rados.0.smithi105.stdout:897: expect (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:51:07.549 INFO:tasks.rados.rados.0.smithi105.stdout:898: set_chunk oid 48 target oid 2024-04-20T02:51:07.599 INFO:tasks.rados.rados.0.smithi105.stdout:898: set_chunk oid 48 offset: 489984 length: 1688576 target oid ee803409e771d29ef947ee706c9a47217f1f7d7269a29b50ff7d49ff9b41e0c0 offset: 0 snap 6 2024-04-20T02:51:07.599 INFO:tasks.rados.rados.0.smithi105.stdout:899: rollback oid 13 current snap is 9 2024-04-20T02:51:07.599 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 13 to 8 2024-04-20T02:51:07.599 INFO:tasks.rados.rados.0.smithi105.stdout:892: finishing rollback tid 0 to smithi10548823-12 2024-04-20T02:51:07.599 INFO:tasks.rados.rados.0.smithi105.stdout:892: finishing rollback tid 1 to smithi10548823-12 2024-04-20T02:51:07.599 INFO:tasks.rados.rados.0.smithi105.stdout:892: finishing rollback tid 2 to smithi10548823-12 2024-04-20T02:51:07.600 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 12 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-04-20T02:51:07.600 INFO:tasks.rados.rados.0.smithi105.stdout:891: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-20T02:51:07.634 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:07.635+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:07.719 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:07.719 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:07.732 INFO:tasks.rados.rados.0.smithi105.stdout:890: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:51:07.733 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:07.734 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:07.735+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:07.835 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:07.836+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:07.904 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:07.904 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:07.912 INFO:tasks.rados.rados.0.smithi105.stdout:897: expect (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:51:07.914 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:07.936 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:07.937+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:08.037 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:08.037+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:08.076 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:08.084 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:08.084 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:08.095 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:08.137 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:08.137+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:08.194 INFO:tasks.rados.rados.0.smithi105.stdout:894: expect (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:51:08.233 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:08.238 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:08.239+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:08.276 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:08.276 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:08.286 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:08.339 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:08.339+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:08.344 INFO:tasks.rados.rados.0.smithi105.stdout:893: finishing write tid 1 to smithi10548823-28 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:893: finishing write tid 2 to smithi10548823-28 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:893: finishing write tid 3 to smithi10548823-28 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:893: finishing write tid 4 to smithi10548823-28 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:893: oid 28 updating version 0 to 71 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:893: oid 28 updating version 71 to 72 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:893: oid 28 updating version 72 to 73 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 73 (ObjNum 138 snap 9 seq_num 138) dirty exists 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:893: left oid 28 (ObjNum 138 snap 9 seq_num 138) 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:890: done (9 left) 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:891: done (8 left) 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:892: done (7 left) 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:893: done (6 left) 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:894: done (5 left) 2024-04-20T02:51:08.345 INFO:tasks.rados.rados.0.smithi105.stdout:895: done (4 left) 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:896: done (3 left) 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:897: done (2 left) 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:900: read oid 42 snap -1 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:900: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:901: write oid 36 current snap is 9 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:898: finishing set_chunk to oid 48 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 48 set_chunk returned error code -95 offset: 489984 length: 1688576 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 0 (ObjNum 82 snap 1 seq_num 82) dirty exists 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:899: finishing rollback tid 0 to smithi10548823-13 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:899: finishing rollback tid 1 to smithi10548823-13 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:899: finishing rollback tid 2 to smithi10548823-13 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 110 (ObjNum 97 snap 4 seq_num 97) dirty exists 2024-04-20T02:51:08.346 INFO:tasks.rados.rados.0.smithi105.stdout:901: seq_num 139 ranges {736173=553965,1867350=468823} 2024-04-20T02:51:08.351 INFO:tasks.rados.rados.0.smithi105.stdout:901: writing smithi10548823-36 from 736173 to 1290138 tid 1 2024-04-20T02:51:08.357 INFO:tasks.rados.rados.0.smithi105.stdout:901: writing smithi10548823-36 from 1867350 to 2336173 tid 2 2024-04-20T02:51:08.358 INFO:tasks.rados.rados.0.smithi105.stdout:898: done (3 left) 2024-04-20T02:51:08.358 INFO:tasks.rados.rados.0.smithi105.stdout:899: done (2 left) 2024-04-20T02:51:08.358 INFO:tasks.rados.rados.0.smithi105.stdout:902: read oid 38 snap 5 2024-04-20T02:51:08.358 INFO:tasks.rados.rados.0.smithi105.stdout:902: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:51:08.358 INFO:tasks.rados.rados.0.smithi105.stdout:903: read oid 18 snap -1 2024-04-20T02:51:08.358 INFO:tasks.rados.rados.0.smithi105.stdout:903: expect (ObjNum 83 snap 1 seq_num 83) 2024-04-20T02:51:08.358 INFO:tasks.rados.rados.0.smithi105.stdout:904: delete oid 10 current snap is 9 2024-04-20T02:51:08.358 INFO:tasks.rados.rados.0.smithi105.stdout:900: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:51:08.428 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.2 to 1 (10000) 2024-04-20T02:51:08.439 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:08.439 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:08.440 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:08.440+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:08.440 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-20T02:51:08.450 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:08.455 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:08.474 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:08.495 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:08.514 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:08.535 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:08.540 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:08.540+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:08.544 INFO:tasks.rados.rados.0.smithi105.stdout:902: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-20T02:51:08.554 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:08.575 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:08.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:08.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:08.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:08.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:08.577 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:08.577 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:08.577 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:08.577 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:08.624 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:08.624 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:08.624 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:08.624 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:08.624 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:08.634 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:08.635 INFO:tasks.rados.rados.0.smithi105.stdout:903: expect (ObjNum 83 snap 1 seq_num 83) 2024-04-20T02:51:08.641 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:08.641+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:08.742 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:08.742+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:08.789 INFO:tasks.rados.rados.0.smithi105.stdout:901: finishing write tid 1 to smithi10548823-36 2024-04-20T02:51:08.789 INFO:tasks.rados.rados.0.smithi105.stdout:901: finishing write tid 2 to smithi10548823-36 2024-04-20T02:51:08.789 INFO:tasks.rados.rados.0.smithi105.stdout:901: finishing write tid 3 to smithi10548823-36 2024-04-20T02:51:08.789 INFO:tasks.rados.rados.0.smithi105.stdout:901: finishing write tid 4 to smithi10548823-36 2024-04-20T02:51:08.789 INFO:tasks.rados.rados.0.smithi105.stdout:901: oid 36 updating version 0 to 114 2024-04-20T02:51:08.789 INFO:tasks.rados.rados.0.smithi105.stdout:901: oid 36 version 114 is already newer than 113 2024-04-20T02:51:08.790 INFO:tasks.rados.rados.0.smithi105.stdout:901: oid 36 version 114 is already newer than 112 2024-04-20T02:51:08.790 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 114 (ObjNum 139 snap 9 seq_num 139) dirty exists 2024-04-20T02:51:08.790 INFO:tasks.rados.rados.0.smithi105.stdout:901: left oid 36 (ObjNum 139 snap 9 seq_num 139) 2024-04-20T02:51:08.790 INFO:tasks.rados.rados.0.smithi105.stdout:900: done (4 left) 2024-04-20T02:51:08.790 INFO:tasks.rados.rados.0.smithi105.stdout:901: done (3 left) 2024-04-20T02:51:08.790 INFO:tasks.rados.rados.0.smithi105.stdout:902: done (2 left) 2024-04-20T02:51:08.790 INFO:tasks.rados.rados.0.smithi105.stdout:903: done (1 left) 2024-04-20T02:51:08.790 INFO:tasks.rados.rados.0.smithi105.stdout:904: done (0 left) 2024-04-20T02:51:08.790 INFO:tasks.rados.rados.0.smithi105.stdout:905: set_chunk oid 7 target oid 2024-04-20T02:51:08.805 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:08.805 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.861579458000001, 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.806 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:08.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:51:08.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:51:08.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:51:08.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:08.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:51:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:51:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:51:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:51:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:51:08.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.854101140000001, 2024-04-20T02:51:08.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:51:08.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:51:08.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1146 4.d 4:b376c20e:::smithi10548823-49:head [write 3437750~125301] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:08.822 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.849533278999999, 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070707487999999999, 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1146 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:08.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955022+0000", 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955179+0000", 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015657 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955188+0000", 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.645e-06 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955195+0000", 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2910000000000004e-06 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978329+0000", 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02313401 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978450+0000", 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121438 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978652+0000", 2024-04-20T02:51:08.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020163700000000001 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984380+0000", 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057282510000000002 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991325+0000", 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069446830000000001 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025680+0000", 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034355284 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025709+0000", 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8781000000000001e-05 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025731+0000", 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1963e-05 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1147 4.d 4:b376c20e:::smithi10548823-49:head [setxattr _header (56) in=7b,truncate 3563051] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.849354942000002, 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070713812000000001, 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1147 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955201+0000", 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:08.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955204+0000", 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.05e-06 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955207+0000", 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1949999999999999e-06 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955210+0000", 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983351+0000", 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028140978000000001 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983488+0000", 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013647 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:08.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983668+0000", 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000180335 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984487+0000", 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081847099999999998 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991393+0000", 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069068569999999998 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025852+0000", 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034458946999999997 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025883+0000", 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0261e-05 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025915+0000", 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2654999999999998e-05 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.849343241, 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.835 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:51:08.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.425914306999999, 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.840 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:51:08.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.847 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.425877423999999, 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:08.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:51:08.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:51:08.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:51:08.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:51:08.853 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.425852791000001, 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.854 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:51:08.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:51:08.858 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.41935938, 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:51:08.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:51:08.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.865 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.413295357999999, 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.866 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:08.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:51:08.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:08.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.871 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.403233593, 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.872 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:08.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.402540989, 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:51:08.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:08.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:51:08.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:51:08.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:51:08.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:51:08.883 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.402529335000001, 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:51:08.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:08.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:51:08.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6964647789999994, 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:51:08.892 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1426 4.1 4:8b927dd6:::smithi10548823-28:head [write 1331027~582033] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6849106169999999, 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068185170000000003, 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1426 2024-04-20T02:51:08.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119645+0000", 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120235+0000", 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059073600000000002 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120249+0000", 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4168000000000001e-05 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120258+0000", 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.2719999999999997e-06 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124047+0000", 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003788839 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124154+0000", 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000107747 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:08.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124307+0000", 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000152998 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148497+0000", 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024189789999999999 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168007+0000", 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019509463000000001 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187765+0000", 2024-04-20T02:51:08.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019758738000000001 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187802+0000", 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7153999999999997e-05 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187831+0000", 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8592999999999999e-05 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1428 4.1 4:8b927dd6:::smithi10548823-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.684257788, 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067836666000000004, 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1428 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120298+0000", 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120301+0000", 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.853e-06 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120308+0000", 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0559999999999999e-06 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120313+0000", 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1680000000000003e-06 2024-04-20T02:51:08.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148583+0000", 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028269882 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148673+0000", 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9922999999999999e-05 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187934+0000", 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039260647000000003 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188031+0000", 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7468999999999998e-05 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188135+0000", 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104126 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6774040630000009, 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:51:08.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:51:08.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:51:08.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.671457749, 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:51:08.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:51:08.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:51:08.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:51:08.914 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6708923079999991, 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.915 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:51:08.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.670866277, 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:08.921 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:08.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:08.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:08.924 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:08.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:08.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:08.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:51:08.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:51:08.926 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.926 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.926 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.926 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:08.926 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:08.926 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:08.928 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:08.930 INFO:tasks.rados.rados.0.smithi105.stdout:905: set_chunk oid 7 offset: 283648 length: 2613760 target oid a1744cc895df5e9d3e624b281fcb9edc57f3f342d1f108d4c38dc778f219a41a offset: 0 snap -1 2024-04-20T02:51:08.930 INFO:tasks.rados.rados.0.smithi105.stdout:906: read oid 1 snap 8 2024-04-20T02:51:08.930 INFO:tasks.rados.rados.0.smithi105.stdout:906: expect (ObjNum 81 snap 1 seq_num 81) 2024-04-20T02:51:08.930 INFO:tasks.rados.rados.0.smithi105.stdout:907: set_chunk oid 45 target oid 2024-04-20T02:51:08.930 INFO:tasks.rados.rados.0.smithi105.stdout:907: set_chunk oid 45 offset: 356864 length: 1556992 target oid 7aea211498f946c134ed12b1f5632511a2f083a0eb2114d1051f38dc7326b113 offset: 0 snap -1 2024-04-20T02:51:08.930 INFO:tasks.rados.rados.0.smithi105.stdout:908: set_chunk oid 10 target oid 2024-04-20T02:51:08.930 INFO:tasks.rados.rados.0.smithi105.stdout:908: set_chunk oid 10 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:909: read oid 29 snap -1 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:909: expect (ObjNum 85 snap 2 seq_num 85) 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:910: write oid 50 current snap is 9 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:905: finishing set_chunk to oid 7 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 7 set_chunk returned error code -95 offset: 283648 length: 2613760 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 0 (ObjNum 105 snap 5 seq_num 105) dirty exists 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:910: seq_num 141 ranges {769787=704463,2267961=101826} 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:910: writing smithi10548823-50 from 769787 to 1474250 tid 1 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:910: writing smithi10548823-50 from 2267961 to 2369787 tid 2 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:905: done (5 left) 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:911: read oid 12 snap -1 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:911: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-20T02:51:08.931 INFO:tasks.rados.rados.0.smithi105.stdout:912: write (excl) oid 31 current snap is 9 2024-04-20T02:51:08.932 INFO:tasks.rados.rados.0.smithi105.stdout:912: seq_num 142 ranges {786594=779712,2326443=60151} 2024-04-20T02:51:08.932 INFO:tasks.rados.rados.0.smithi105.stdout:912: writing smithi10548823-31 from 786594 to 1566306 tid 1 2024-04-20T02:51:08.932 INFO:tasks.rados.rados.0.smithi105.stdout:912: writing smithi10548823-31 from 2326443 to 2386594 tid 2 2024-04-20T02:51:08.932 INFO:tasks.rados.rados.0.smithi105.stdout:913: tier_promote oid 14 2024-04-20T02:51:08.932 INFO:tasks.rados.rados.0.smithi105.stdout:906: expect (ObjNum 81 snap 1 seq_num 81) 2024-04-20T02:51:08.932 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:08.928+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:09.028 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:09.029+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:09.075 INFO:tasks.rados.rados.0.smithi105.stdout:907: finishing set_chunk to oid 45 2024-04-20T02:51:09.075 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 356864 length: 1556992 2024-04-20T02:51:09.075 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:51:09.075 INFO:tasks.rados.rados.0.smithi105.stdout:908: finishing set_chunk to oid 10 2024-04-20T02:51:09.075 INFO:tasks.rados.rados.0.smithi105.stdout:908: got expected ERANGE (src dne) 2024-04-20T02:51:09.075 INFO:tasks.rados.rados.0.smithi105.stdout:906: done (7 left) 2024-04-20T02:51:09.075 INFO:tasks.rados.rados.0.smithi105.stdout:907: done (6 left) 2024-04-20T02:51:09.075 INFO:tasks.rados.rados.0.smithi105.stdout:908: done (5 left) 2024-04-20T02:51:09.075 INFO:tasks.rados.rados.0.smithi105.stdout:914: write oid 48 current snap is 9 2024-04-20T02:51:09.075 INFO:tasks.rados.rados.0.smithi105.stdout:909: expect (ObjNum 85 snap 2 seq_num 85) 2024-04-20T02:51:09.097 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:09.097 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:09.097 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:09.097 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:09.107 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:09.133 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:51:09.151 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:09.152+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:09.230 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:09.230+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:09.235 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:09.235 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:09.235 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:09.235 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:09.235 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:09.245 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:09.275 INFO:tasks.rados.rados.0.smithi105.stdout:911: expect (ObjNum 65 snap 0 seq_num 65) 2024-04-20T02:51:09.331 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:09.332+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:09.416 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.680719099000001, 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:09.417 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:51:09.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:51:09.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:51:09.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:09.427 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.649430037999998, 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:09.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:09.433 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.644739582, 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:09.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:51:09.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:09.439 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.637983113000001, 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:09.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:51:09.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:51:09.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:51:09.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:51:09.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:51:09.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:51:09.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:51:09.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.628167602000001, 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:09.446 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:51:09.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:51:09.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.62712367, 2024-04-20T02:51:09.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:51:09.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:51:09.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:51:09.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:51:09.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.62710251, 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.459 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:09.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:51:09.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:51:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.419531929000001, 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:09.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:51:09.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:51:09.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.405337284000002, 2024-04-20T02:51:09.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:51:09.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.399370027, 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:51:09.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.390964023999999, 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.477 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:09.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.481 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.287331983000001, 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.056735672999999, 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:51:09.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:51:09.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.029791524, 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:51:09.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.004433256999999, 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:51:09.494 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:51:09.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 26.976292813000001, 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:51:09.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:51:09.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 26.938924208, 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.503 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:51:09.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.507 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 26.938627284999999, 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:51:09.511 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 26.933544167000001, 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:51:09.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 26.906541608000001, 2024-04-20T02:51:09.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:51:09.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:51:09.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.520 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:09.521 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:09.521 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:09.523 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:09.523 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:09.522+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:09.524 INFO:tasks.rados.rados.0.smithi105.stdout:910: finishing write tid 1 to smithi10548823-50 2024-04-20T02:51:09.524 INFO:tasks.rados.rados.0.smithi105.stdout:910: finishing write tid 2 to smithi10548823-50 2024-04-20T02:51:09.524 INFO:tasks.rados.rados.0.smithi105.stdout:910: finishing write tid 3 to smithi10548823-50 2024-04-20T02:51:09.524 INFO:tasks.rados.rados.0.smithi105.stdout:914: seq_num 143 ranges {403401=454961,1584925=506039} 2024-04-20T02:51:09.524 INFO:tasks.rados.rados.0.smithi105.stdout:914: writing smithi10548823-48 from 403401 to 858362 tid 1 2024-04-20T02:51:09.524 INFO:tasks.rados.rados.0.smithi105.stdout:914: writing smithi10548823-48 from 1584925 to 2090964 tid 2 2024-04-20T02:51:09.524 INFO:tasks.rados.rados.0.smithi105.stdout:909: done (5 left) 2024-04-20T02:51:09.524 INFO:tasks.rados.rados.0.smithi105.stdout:911: done (4 left) 2024-04-20T02:51:09.524 INFO:tasks.rados.rados.0.smithi105.stdout:915: tier_evict oid 19 2024-04-20T02:51:09.525 INFO:tasks.rados.rados.0.smithi105.stdout:915: tier_evict oid 19 snap -1 2024-04-20T02:51:09.525 INFO:tasks.rados.rados.0.smithi105.stdout:916: set_chunk oid 41 target oid 2024-04-20T02:51:09.525 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.3 to 1 (10000) 2024-04-20T02:51:09.526 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-20T02:51:09.527 INFO:tasks.rados.rados.0.smithi105.stdout:916: set_chunk oid 41 offset: 321024 length: 2962944 target oid 599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c offset: 0 snap -1 2024-04-20T02:51:09.527 INFO:tasks.rados.rados.0.smithi105.stdout:917: read oid 6 snap -1 2024-04-20T02:51:09.527 INFO:tasks.rados.rados.0.smithi105.stdout:917: expect deleted 2024-04-20T02:51:09.527 INFO:tasks.rados.rados.0.smithi105.stdout:918: write (excl) oid 5 current snap is 9 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:910: finishing write tid 4 to smithi10548823-50 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:910: oid 50 updating version 0 to 30 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:910: oid 50 version 30 is already newer than 28 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:910: oid 50 version 30 is already newer than 29 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 30 (ObjNum 141 snap 9 seq_num 141) dirty exists 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:910: left oid 50 (ObjNum 141 snap 9 seq_num 141) 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:913: got (0) Success 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 41 (ObjNum 99 snap 5 seq_num 99) dirty exists 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:912: finishing write tid 1 to smithi10548823-31 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:912: finishing write tid 2 to smithi10548823-31 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:918: seq_num 144 ranges {420208=530210,1527054=766050} 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:918: writing smithi10548823-5 from 420208 to 950418 tid 1 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:918: writing smithi10548823-5 from 1527054 to 2293104 tid 2 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:910: done (7 left) 2024-04-20T02:51:09.528 INFO:tasks.rados.rados.0.smithi105.stdout:913: done (6 left) 2024-04-20T02:51:09.529 INFO:tasks.rados.rados.0.smithi105.stdout:919: read oid 27 snap -1 2024-04-20T02:51:09.529 INFO:tasks.rados.rados.0.smithi105.stdout:919: expect (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:51:09.529 INFO:tasks.rados.rados.0.smithi105.stdout:920: set_chunk oid 25 target oid 2024-04-20T02:51:09.542 INFO:tasks.rados.rados.0.smithi105.stdout:920: set_chunk oid 25 offset: 842240 length: 610816 target oid dca41305e4ccb2ee245f1b6b54b604d146732990cffae806456632218683be86 offset: 0 snap -1 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:921: snap_remove snap 5 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:912: finishing write tid 3 to smithi10548823-31 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:912: finishing write tid 4 to smithi10548823-31 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:912: oid 31 updating version 0 to 63 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:912: oid 31 updating version 63 to 64 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:912: oid 31 version 64 is already newer than 62 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 64 (ObjNum 142 snap 9 seq_num 142) dirty exists 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:912: left oid 31 (ObjNum 142 snap 9 seq_num 142) 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:915: got (22) Invalid argument 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:914: finishing write tid 1 to smithi10548823-48 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:914: finishing write tid 2 to smithi10548823-48 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:914: finishing write tid 3 to smithi10548823-48 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:914: finishing write tid 4 to smithi10548823-48 2024-04-20T02:51:09.543 INFO:tasks.rados.rados.0.smithi105.stdout:914: oid 48 updating version 0 to 43 2024-04-20T02:51:09.544 INFO:tasks.rados.rados.0.smithi105.stdout:914: oid 48 version 43 is already newer than 41 2024-04-20T02:51:09.544 INFO:tasks.rados.rados.0.smithi105.stdout:914: oid 48 version 43 is already newer than 42 2024-04-20T02:51:09.544 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 43 (ObjNum 143 snap 9 seq_num 143) dirty exists 2024-04-20T02:51:09.544 INFO:tasks.rados.rados.0.smithi105.stdout:914: left oid 48 (ObjNum 143 snap 9 seq_num 143) 2024-04-20T02:51:09.623 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:09.623+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:09.691 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:09.691 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:09.691 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:09.691 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:09.702 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:09.723 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:09.724+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:09.824 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:09.825+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:09.870 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:09.870 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:09.870 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:09.870 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:09.870 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:09.880 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:09.924 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:09.925+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:10.011 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:10.011 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:10.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:10.011 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:10.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 50.406538992000002, 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:51:10.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:51:10.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:51:10.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:51:10.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:51:10.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:51:10.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:51:10.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:51:10.018 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.019 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.019 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.021 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:29 4.8 4:1c68e542:::smithi10548823-8:head [write 517649~526743,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:10.021 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:51:10.021 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.653664361000001, 2024-04-20T02:51:10.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031979782999999998, 2024-04-20T02:51:10.021 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.021 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.021 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.021 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 29 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356789+0000", 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.356788+0000", 2024-04-20T02:51:10.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357534+0000", 2024-04-20T02:51:10.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074630400000000004 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357547+0000", 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2697000000000001e-05 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357556+0000", 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8040000000000007e-06 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357617+0000", 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1038999999999996e-05 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357754+0000", 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013660500000000001 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.357853+0000", 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9327000000000006e-05 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388575+0000", 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030721678999999998 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388623+0000", 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878999999999997e-05 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388660+0000", 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7002000000000002e-05 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388713+0000", 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2608999999999997e-05 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388769+0000", 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5896999999999997e-05 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:30 4.8 4:1c68e542:::smithi10548823-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:10.027 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.646386803999999, 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033039351000000002, 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 30 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:10.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:10.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364066+0000", 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364312+0000", 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000246708 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364328+0000", 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5376999999999999e-05 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364339+0000", 2024-04-20T02:51:10.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.092e-05 2024-04-20T02:51:10.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375692+0000", 2024-04-20T02:51:10.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011353103999999999 2024-04-20T02:51:10.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375805+0000", 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000113414 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375970+0000", 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016508399999999999 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388615+0000", 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012645145 2024-04-20T02:51:10.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391156+0000", 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025406410000000002 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396990+0000", 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0058340390000000001 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397042+0000", 2024-04-20T02:51:10.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2454000000000003e-05 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397106+0000", 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3632000000000003e-05 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:31 4.8 4:1c68e542:::smithi10548823-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.646100107999999, 2024-04-20T02:51:10.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032910542000000001, 2024-04-20T02:51:10.037 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.037 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.037 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.037 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.037 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.037 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 31 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364357+0000", 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1160000000000001e-06 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364363+0000", 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1680000000000001e-06 2024-04-20T02:51:10.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364367+0000", 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8750000000000002e-06 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388814+0000", 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024447437999999998 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388914+0000", 2024-04-20T02:51:10.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9557999999999998e-05 2024-04-20T02:51:10.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:10.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389062+0000", 2024-04-20T02:51:10.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014799600000000001 2024-04-20T02:51:10.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:10.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391160+0000", 2024-04-20T02:51:10.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020979760000000001 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396038+0000", 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048784689999999999 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397155+0000", 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001116286 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397208+0000", 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3273000000000003e-05 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397264+0000", 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5962999999999999e-05 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:32 4.8 4:1c68e542:::smithi10548823-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.646081393999999, 2024-04-20T02:51:10.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033121840999999999, 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 32 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364375+0000", 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1149999999999998e-06 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364382+0000", 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4259999999999997e-06 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364387+0000", 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6129999999999997e-06 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389813+0000", 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025426621 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389905+0000", 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2011999999999999e-05 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397299+0000", 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007393897 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397395+0000", 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5293999999999999e-05 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397494+0000", 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9289000000000004e-05 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.642672593, 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:51:10.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:51:10.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.529369551999999, 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:66 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:10.058 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.524011664, 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033365526, 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 66 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486438+0000", 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486778+0000", 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000339709 2024-04-20T02:51:10.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486798+0000", 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0462000000000002e-05 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486812+0000", 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3589e-05 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496889+0000", 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010077477 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496978+0000", 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8205e-05 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.497125+0000", 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000147665 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518444+0000", 2024-04-20T02:51:10.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.021318713 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519407+0000", 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00096312999999999998 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519727+0000", 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031953500000000002 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519764+0000", 2024-04-20T02:51:10.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7406000000000001e-05 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519807+0000", 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3402000000000002e-05 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:67 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2268912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.523626480000001, 2024-04-20T02:51:10.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033307638000000001, 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 67 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486826+0000", 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486831+0000", 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0189999999999997e-06 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486839+0000", 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4710000000000001e-06 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486844+0000", 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.57e-06 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509240+0000", 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022395241 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509338+0000", 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8034000000000004e-05 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509498+0000", 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015993999999999999 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518471+0000", 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0089732730000000004 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519859+0000", 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013877130000000001 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520042+0000", 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182981 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520084+0000", 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2518999999999997e-05 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520135+0000", 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0491000000000002e-05 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.523604034999998, 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.070 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:51:10.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:10.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:161 4.3 e29/27 4:c32272c0:::smithi10548823-41:head v 29'16, mlcod=29'14)", 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.176167943999999, 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031990785000000001, 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834285+0000", 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835168+0000", 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00088287800000000003 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835180+0000", 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2340999999999999e-05 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835187+0000", 2024-04-20T02:51:10.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7830000000000001e-06 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835227+0000", 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0754000000000003e-05 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835243+0000", 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5607000000000002e-05 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866234+0000", 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030990891 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866276+0000", 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2469999999999998e-05 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.169583899999999, 2024-04-20T02:51:10.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:51:10.083 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.159220295000001, 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:51:10.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.151135816, 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.091 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:981 4.7 e32/27 4:e9876a63:::smithi10548823-30:head v 32'19, mlcod=30'17)", 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.421083341999999, 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033092985999999998, 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589369+0000", 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590134+0000", 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076479 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590148+0000", 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3679999999999999e-05 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590157+0000", 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590209+0000", 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2237999999999997e-05 2024-04-20T02:51:10.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590235+0000", 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6251e-05 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622430+0000", 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032194953999999998 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622463+0000", 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2780000000000001e-05 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.095 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1230 4.5 e38/22 4:a94dc531:::smithi10548823-32:head v 38'11, mlcod=29'8)", 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.597937944, 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033312785999999997, 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412515+0000", 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412827+0000", 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031200999999999999 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412835+0000", 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.61e-06 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412842+0000", 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6990000000000003e-06 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412921+0000", 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8380000000000005e-05 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412944+0000", 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3609e-05 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445795+0000", 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032850279000000003 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445828+0000", 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3673000000000001e-05 2024-04-20T02:51:10.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8695775910000005, 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:51:10.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:51:10.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:10.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:51:10.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:10.105 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8644404810000008, 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:10.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:51:10.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:51:10.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:51:10.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8641674689999999, 2024-04-20T02:51:10.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8641380840000004, 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:10.117 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:51:10.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:51:10.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:51:10.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:51:10.122 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.122 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.122 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.122 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.122 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.122 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:10.123 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:10.124 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:10.124+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:10.224 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:10.225+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:10.292 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:10.292 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:10.292 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:10.292 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:10.302 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:10.325 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:10.326+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:10.426 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:10.426+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:10.445 INFO:tasks.rados.rados.0.smithi105.stdout:912: done (8 left) 2024-04-20T02:51:10.445 INFO:tasks.rados.rados.0.smithi105.stdout:914: done (7 left) 2024-04-20T02:51:10.445 INFO:tasks.rados.rados.0.smithi105.stdout:915: done (6 left) 2024-04-20T02:51:10.446 INFO:tasks.rados.rados.0.smithi105.stdout:917: done (5 left) 2024-04-20T02:51:10.446 INFO:tasks.rados.rados.0.smithi105.stdout:921: done (4 left) 2024-04-20T02:51:10.446 INFO:tasks.rados.rados.0.smithi105.stdout:922: write oid 34 current snap is 9 2024-04-20T02:51:10.446 INFO:tasks.rados.rados.0.smithi105.stdout:922: seq_num 145 ranges {437015=721812,1701889=509708} 2024-04-20T02:51:10.449 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:51:10.452 INFO:tasks.rados.rados.0.smithi105.stdout:922: writing smithi10548823-34 from 437015 to 1158827 tid 1 2024-04-20T02:51:10.459 INFO:tasks.rados.rados.0.smithi105.stdout:922: writing smithi10548823-34 from 1701889 to 2211597 tid 2 2024-04-20T02:51:10.459 INFO:tasks.rados.rados.0.smithi105.stdout:923: tier_evict oid 45 2024-04-20T02:51:10.459 INFO:tasks.rados.rados.0.smithi105.stdout:923: tier_evict oid 45 snap -1 2024-04-20T02:51:10.459 INFO:tasks.rados.rados.0.smithi105.stdout:924: read oid 43 snap -1 2024-04-20T02:51:10.459 INFO:tasks.rados.rados.0.smithi105.stdout:924: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:51:10.459 INFO:tasks.rados.rados.0.smithi105.stdout:925: read oid 10 snap -1 2024-04-20T02:51:10.459 INFO:tasks.rados.rados.0.smithi105.stdout:925: expect deleted 2024-04-20T02:51:10.460 INFO:tasks.rados.rados.0.smithi105.stdout:926: snap_remove snap 8 2024-04-20T02:51:10.461 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.4 to 1 (10000) 2024-04-20T02:51:10.477 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-20T02:51:10.483 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:10.483 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:10.495 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:10.561 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:10.562+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:10.589 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:10.627 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:10.627+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:10.666 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:10.666 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.3 46/44 [PushOp(4:cfd1e4d4:::smithi10548823-9:head, version: 45'100, data_included: [1817887~282988], data_size: 282988, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:cfd1e4d4:::smithi10548823-9:head@45'100, size: 2100875, copy_subset: [0~2100875], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2100875, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1817887, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:03.670012+0000", 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 6.996039101, 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0085824019999999994, 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670012+0000", 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670012+0000", 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670011+0000", 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670238+0000", 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022708599999999999 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670252+0000", 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3485e-05 2024-04-20T02:51:10.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670264+0000", 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2007999999999999e-05 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670313+0000", 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8925000000000003e-05 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670322+0000", 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8049999999999996e-06 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.678594+0000", 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0082727180000000001 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1528 3.2 e48/47 3:7333b787:::49dfa3a36ea566ece109be00940c25f90e74796f7ebe644247ba8ecd48263ae5:head v 48'95, mlcod=0'0)", 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:10.671 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.2436156470000004, 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088605330000000003, 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423048+0000", 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00061281499999999995 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423060+0000", 2024-04-20T02:51:10.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1644000000000001e-05 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423068+0000", 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7460000000000006e-06 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423108+0000", 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.977e-05 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423134+0000", 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5599e-05 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431261+0000", 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081278059999999996 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431296+0000", 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4413999999999999e-05 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.675 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(3.2 48/47 [PushOp(3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head, version: 45'92, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head@45'92, size: 524288, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.430426+0000", 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.2356253449999999, 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00060689100000000003, 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.430426+0000", 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.430426+0000", 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.430425+0000", 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.430443+0000", 2024-04-20T02:51:10.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8525e-05 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.430463+0000", 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9504000000000002e-05 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.430497+0000", 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4307000000000002e-05 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.430834+0000", 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00033722400000000001 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.430849+0000", 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4449e-05 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431033+0000", 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018377899999999999 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.679 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(3.2 48/47 [PushOp(3:74db4bdf:::5e1e28d63161cc349a212a2b469bd984da8549c81d969fef5410dc83e2078908:head, version: 45'93, data_included: [196608~72001], data_size: 72001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:74db4bdf:::5e1e28d63161cc349a212a2b469bd984da8549c81d969fef5410dc83e2078908:head@45'93, size: 268609, copy_subset: [0~268609], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:268609, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.431434+0000", 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.2346173929999997, 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081109300000000004, 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431434+0000", 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431434+0000", 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431433+0000", 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431498+0000", 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4622000000000006e-05 2024-04-20T02:51:10.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431511+0000", 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3752e-05 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431523+0000", 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2062000000000001e-05 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431585+0000", 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1724999999999996e-05 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431597+0000", 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2145e-05 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432245+0000", 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00064730400000000002 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.683 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(3.2 48/47 [PushOp(3:556eb17c:::3c9812f1a7ee9f26a1175ea4fab5c703e0d495f552e39ebbe4874fc5964ce285:head, version: 45'94, data_included: [0~270207], data_size: 270207, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:556eb17c:::3c9812f1a7ee9f26a1175ea4fab5c703e0d495f552e39ebbe4874fc5964ce285:head@45'94, size: 270207, copy_subset: [0~270207], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:270207, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.432273+0000", 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.2337783059999996, 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0067461539999999999, 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432273+0000", 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432273+0000", 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432272+0000", 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432702+0000", 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00042944400000000002 2024-04-20T02:51:10.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432715+0000", 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3135e-05 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432725+0000", 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0767999999999999e-05 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432765+0000", 2024-04-20T02:51:10.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9779e-05 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432779+0000", 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4266e-05 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439019+0000", 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0062393290000000001 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:152 3.2 e48/47 3:54f0c00e:::2528189ec1c891f179c539fa23af2c87b9e8d2c19f70e70fac8934d3cc51ccee:head v 48'96, mlcod=0'0)", 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.2331447210000004, 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020196797999999998, 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433091+0000", 2024-04-20T02:51:10.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000185589 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433102+0000", 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433108+0000", 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5660000000000003e-06 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439137+0000", 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060284329999999997 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439166+0000", 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9278e-05 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453075+0000", 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013909163 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453103+0000", 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7725999999999998e-05 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:154 3.2 e48/47 3:7d72d9ce:::7b4ada8765f224bb4a516820a83f51a402b4ee671c5f45bb3fbe5e6656cf1f14:head v 48'97, mlcod=42'91)", 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.229239089, 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016320239, 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.692 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436811+0000", 2024-04-20T02:51:10.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437600+0000", 2024-04-20T02:51:10.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078939699999999995 2024-04-20T02:51:10.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437613+0000", 2024-04-20T02:51:10.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2588e-05 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437624+0000", 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0542e-05 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442541+0000", 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049178809999999998 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442565+0000", 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3558e-05 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453110+0000", 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010545172 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453132+0000", 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1929e-05 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:144 3.2 e48/47 3:7333b787:::49dfa3a36ea566ece109be00940c25f90e74796f7ebe644247ba8ecd48263ae5:head v 48'98, mlcod=42'91)", 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.438561+0000", 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.2274900420000003, 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014604997999999999, 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438561+0000", 2024-04-20T02:51:10.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438561+0000", 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438560+0000", 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438568+0000", 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3380000000000005e-06 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438583+0000", 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4284000000000001e-05 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438591+0000", 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3569999999999998e-06 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443406+0000", 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048148610000000001 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443435+0000", 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8915999999999999e-05 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453135+0000", 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0097005549999999996 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453166+0000", 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0516999999999999e-05 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:155 3.2 e48/47 3:715fab8c:::b945a2927747be608263bdbe1fe814f665c73382cb3375555da5aa95a92567ff:head v 48'99, mlcod=42'91)", 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.2264520340000002, 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017321785999999999, 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.701 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439598+0000", 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439954+0000", 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00035586299999999999 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439970+0000", 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5889e-05 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439981+0000", 2024-04-20T02:51:10.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0754000000000001e-05 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443643+0000", 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036621259999999999 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443671+0000", 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8289e-05 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456885+0000", 2024-04-20T02:51:10.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013213964999999999 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456921+0000", 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1529 4.8 e48/44 4:1afa5d2b:::smithi10548823-17:head v 48'47, mlcod=0'0)", 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.455478+0000", 2024-04-20T02:51:10.705 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.210573084, 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003386975, 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455478+0000", 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455478+0000", 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455478+0000", 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455482+0000", 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6349999999999997e-06 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455492+0000", 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5550000000000008e-06 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455500+0000", 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9189999999999999e-06 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455600+0000", 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100413 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455623+0000", 2024-04-20T02:51:10.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3183000000000001e-05 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.458809+0000", 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003185818 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.458865+0000", 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5761999999999999e-05 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1549 4.3 e48/47 4:c07c6daa:::smithi10548823-7:head v 48'114, mlcod=0'0)", 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.491194+0000", 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1748573990000004, 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029922030000000001, 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.491194+0000", 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.491194+0000", 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.491193+0000", 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.491203+0000", 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0232e-05 2024-04-20T02:51:10.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.491217+0000", 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4504e-05 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.491224+0000", 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1899999999999998e-06 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.491267+0000", 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2301e-05 2024-04-20T02:51:10.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.491289+0000", 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2232e-05 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494160+0000", 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0028711969999999998 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494186+0000", 2024-04-20T02:51:10.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5565e-05 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1550 4.3 e48/47 4:c07c6daa:::smithi10548823-7:head v 48'115, mlcod=0'0)", 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.492514+0000", 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1735371209999998, 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020748659999999999, 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.492514+0000", 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.492514+0000", 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.492513+0000", 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.492519+0000", 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5779999999999998e-06 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.492528+0000", 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0780000000000002e-06 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.492535+0000", 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3810000000000001e-06 2024-04-20T02:51:10.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.492569+0000", 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3229000000000003e-05 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.492590+0000", 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0999e-05 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494561+0000", 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019711799999999999 2024-04-20T02:51:10.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494589+0000", 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.796e-05 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1551 4.3 e48/47 4:c07c6daa:::smithi10548823-7:head v 48'116, mlcod=0'0)", 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.494365+0000", 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1716860760000003, 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0028661860000000002, 2024-04-20T02:51:10.718 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494365+0000", 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494365+0000", 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494364+0000", 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494372+0000", 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2599999999999999e-06 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494379+0000", 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2339999999999997e-06 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494385+0000", 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3609999999999997e-06 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494422+0000", 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6538000000000002e-05 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.494442+0000", 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0086999999999999e-05 2024-04-20T02:51:10.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.497205+0000", 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0027629299999999998 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.497231+0000", 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6271e-05 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1635 3.2 e51/49 3:67667c16:::7aea211498f946c134ed12b1f5632511a2f083a0eb2114d1051f38dc7326b113:head v 51'100, mlcod=0'0)", 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.818561087, 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019392829, 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847489+0000", 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850819+0000", 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033292069999999998 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850831+0000", 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2320000000000001e-05 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850838+0000", 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2980000000000003e-06 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850874+0000", 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5800999999999998e-05 2024-04-20T02:51:10.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850890+0000", 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5536000000000002e-05 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866849+0000", 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015959346999999999 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866883+0000", 2024-04-20T02:51:10.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3776999999999999e-05 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.2174128479999999, 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:51:10.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:51:10.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:159 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'102, mlcod=51'100)", 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.502505+0000", 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.163545609, 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001305011, 2024-04-20T02:51:10.731 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.502505+0000", 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.502505+0000", 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.502504+0000", 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.502522+0000", 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7842999999999998e-05 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.502545+0000", 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3376000000000001e-05 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.502559+0000", 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.438e-05 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.502644+0000", 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4929e-05 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.502682+0000", 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7481999999999998e-05 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.503778+0000", 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001096182 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.503810+0000", 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2361999999999999e-05 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.735 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1660 4.3 e52/47 4:c32272c0:::smithi10548823-41:head v 52'127, mlcod=49'124)", 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.504357+0000", 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.1616934699999999, 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00049206200000000003, 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.504357+0000", 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.504357+0000", 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.504357+0000", 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.504361+0000", 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8099999999999999e-06 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.504371+0000", 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0163000000000001e-05 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.504376+0000", 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0340000000000004e-06 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.504414+0000", 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7790999999999998e-05 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.504427+0000", 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2964e-05 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.504831+0000", 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040369499999999998 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.504850+0000", 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8839e-05 2024-04-20T02:51:10.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1664 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'77, mlcod=50'73)", 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.1563338649999999, 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012476671999999999, 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:10.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509983+0000", 2024-04-20T02:51:10.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00026571599999999999 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509992+0000", 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509999+0000", 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1450000000000002e-06 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.510050+0000", 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0562000000000001e-05 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.510064+0000", 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4586000000000001e-05 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.522152+0000", 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012087736 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.522194+0000", 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1851999999999998e-05 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1665 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'78, mlcod=50'73)", 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.1428129949999999, 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025918594, 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523236+0000", 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523658+0000", 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000422082 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523676+0000", 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7716000000000001e-05 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523686+0000", 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9790000000000005e-06 2024-04-20T02:51:10.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523734+0000", 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7579e-05 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523759+0000", 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5593e-05 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549122+0000", 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025363087999999999 2024-04-20T02:51:10.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549157+0000", 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4242999999999997e-05 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1666 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'79, mlcod=50'75)", 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.548111+0000", 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.1179395510000001, 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011902480000000001, 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:10.748 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.548111+0000", 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.548111+0000", 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.548110+0000", 2024-04-20T02:51:10.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.548120+0000", 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.548131+0000", 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1691999999999999e-05 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.548139+0000", 2024-04-20T02:51:10.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9789999999999993e-06 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.548310+0000", 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000170669 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.548332+0000", 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1746000000000002e-05 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549282+0000", 2024-04-20T02:51:10.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094994800000000005 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549302+0000", 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0049000000000001e-05 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:10.752 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:10.753 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:10.754 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:10.754+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:10.754 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:10.758 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:10.781 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:10.801 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:10.821 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:10.854 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:10.855+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:10.886 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:10.886 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:10.896 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:10.981 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:10.982+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:11.042 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:11.056 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:11.056+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:11.062 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:11.064 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:11.064 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:11.075 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:11.083 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:11.084 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:11.085 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:11.085 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:11.085 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:11.085 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:11.085 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:11.085 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:11.085 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:11.156 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:11.157+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:11.214 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:11.214 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:11.224 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:11.257 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:11.258+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:11.353 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:11.353 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:11.358 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:11.359+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:11.363 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:11.447 INFO:tasks.rados.rados.0.smithi105.stdout:926: done (8 left) 2024-04-20T02:51:11.447 INFO:tasks.rados.rados.0.smithi105.stdout:927: read oid 28 snap -1 2024-04-20T02:51:11.447 INFO:tasks.rados.rados.0.smithi105.stdout:927: expect (ObjNum 138 snap 9 seq_num 138) 2024-04-20T02:51:11.447 INFO:tasks.rados.rados.0.smithi105.stdout:928: read oid 21 snap -1 2024-04-20T02:51:11.447 INFO:tasks.rados.rados.0.smithi105.stdout:928: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-20T02:51:11.447 INFO:tasks.rados.rados.0.smithi105.stdout:929: set_chunk oid 30 target oid 2024-04-20T02:51:11.454 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.6 to 1 (10000) 2024-04-20T02:51:11.459 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:11.460+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:11.473 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:929: set_chunk oid 30 offset: 1018368 length: 2339328 target oid 6c183756a10f3bdc3eedca02e23f46de829bb73a1c80733316ef3f52b4866620 offset: 0 snap 6 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:930: read oid 48 snap -1 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:930: expect (ObjNum 143 snap 9 seq_num 143) 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:931: read oid 1 snap 6 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:931: expect (ObjNum 81 snap 1 seq_num 81) 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:916: finishing set_chunk to oid 41 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 112 (ObjNum 87 snap 2 seq_num 87) dirty exists 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:932: read oid 35 snap -1 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:932: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:916: done (13 left) 2024-04-20T02:51:11.535 INFO:tasks.rados.rados.0.smithi105.stdout:933: read oid 39 snap -1 2024-04-20T02:51:11.536 INFO:tasks.rados.rados.0.smithi105.stdout:933: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:11.536 INFO:tasks.rados.rados.0.smithi105.stdout:934: tier_evict oid 26 2024-04-20T02:51:11.536 INFO:tasks.rados.rados.0.smithi105.stdout:934: tier_evict oid 26 snap -1 2024-04-20T02:51:11.536 INFO:tasks.rados.rados.0.smithi105.stdout:919: expect (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:51:11.560 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:11.561+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:11.574 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:11.575 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:11.586 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:11.661 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:11.662+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:11.744 INFO:tasks.rados.rados.0.smithi105.stdout:920: finishing set_chunk to oid 25 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 28 (ObjNum 70 snap 0 seq_num 70) dirty exists 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:918: finishing write tid 1 to smithi10548823-5 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:918: finishing write tid 2 to smithi10548823-5 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:919: done (14 left) 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:920: done (13 left) 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:935: read oid 38 snap -1 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:935: expect deleted 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:936: read oid 44 snap -1 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:936: expect (ObjNum 135 snap 9 seq_num 135) 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:937: snap_create 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:918: finishing write tid 3 to smithi10548823-5 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:918: finishing write tid 4 to smithi10548823-5 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:918: oid 5 updating version 0 to 78 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:918: oid 5 updating version 78 to 79 2024-04-20T02:51:11.745 INFO:tasks.rados.rados.0.smithi105.stdout:918: oid 5 version 79 is already newer than 77 2024-04-20T02:51:11.746 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 79 (ObjNum 144 snap 9 seq_num 144) dirty exists 2024-04-20T02:51:11.746 INFO:tasks.rados.rados.0.smithi105.stdout:918: left oid 5 (ObjNum 144 snap 9 seq_num 144) 2024-04-20T02:51:11.746 INFO:tasks.rados.rados.0.smithi105.stdout:924: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:51:11.757 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:11.758 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:11.762 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:11.763+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:11.768 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:11.836 INFO:tasks.rados.rados.0.smithi105.stdout:923: got (22) Invalid argument 2024-04-20T02:51:11.836 INFO:tasks.rados.rados.0.smithi105.stdout:922: finishing write tid 1 to smithi10548823-34 2024-04-20T02:51:11.836 INFO:tasks.rados.rados.0.smithi105.stdout:922: finishing write tid 2 to smithi10548823-34 2024-04-20T02:51:11.836 INFO:tasks.rados.rados.0.smithi105.stdout:922: finishing write tid 3 to smithi10548823-34 2024-04-20T02:51:11.836 INFO:tasks.rados.rados.0.smithi105.stdout:922: finishing write tid 4 to smithi10548823-34 2024-04-20T02:51:11.837 INFO:tasks.rados.rados.0.smithi105.stdout:922: oid 34 updating version 0 to 128 2024-04-20T02:51:11.837 INFO:tasks.rados.rados.0.smithi105.stdout:922: oid 34 updating version 128 to 129 2024-04-20T02:51:11.837 INFO:tasks.rados.rados.0.smithi105.stdout:922: oid 34 updating version 129 to 130 2024-04-20T02:51:11.837 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 130 (ObjNum 145 snap 9 seq_num 145) dirty exists 2024-04-20T02:51:11.837 INFO:tasks.rados.rados.0.smithi105.stdout:922: left oid 34 (ObjNum 145 snap 9 seq_num 145) 2024-04-20T02:51:11.837 INFO:tasks.rados.rados.0.smithi105.stdout:928: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-20T02:51:11.863 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:11.864+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:11.898 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:11.898 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:11.908 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:11.918 INFO:tasks.rados.rados.0.smithi105.stdout:927: expect (ObjNum 138 snap 9 seq_num 138) 2024-04-20T02:51:11.964 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:11.965+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:12.036 INFO:tasks.rados.rados.0.smithi105.stdout:929: finishing set_chunk to oid 30 2024-04-20T02:51:12.037 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 1018368 length: 2339328 2024-04-20T02:51:12.037 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 107 snap 5 seq_num 107) dirty exists 2024-04-20T02:51:12.037 INFO:tasks.rados.rados.0.smithi105.stdout:934: got (22) Invalid argument 2024-04-20T02:51:12.037 INFO:tasks.rados.rados.0.smithi105.stdout:933: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:12.048 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:12.048 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:12.058 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:12.065 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:12.066+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:12.105 INFO:tasks.rados.rados.0.smithi105.stdout:932: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:51:12.166 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:12.167+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:12.175 INFO:tasks.rados.rados.0.smithi105.stdout:931: expect (ObjNum 81 snap 1 seq_num 81) 2024-04-20T02:51:12.188 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:12.188 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:12.199 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:12.267 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:12.268+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:12.329 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:12.329 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:12.340 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:12.364 INFO:tasks.rados.rados.0.smithi105.stdout:930: expect (ObjNum 143 snap 9 seq_num 143) 2024-04-20T02:51:12.408 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:12.409+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:12.459 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.7 to 1 (10000) 2024-04-20T02:51:12.468 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:12.469+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:12.468 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:51:12.475 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:918: done (15 left) 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:922: done (14 left) 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:923: done (13 left) 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:924: done (12 left) 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:925: done (11 left) 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:927: done (10 left) 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:928: done (9 left) 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:929: done (8 left) 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:930: done (7 left) 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:931: done (6 left) 2024-04-20T02:51:12.503 INFO:tasks.rados.rados.0.smithi105.stdout:932: done (5 left) 2024-04-20T02:51:12.504 INFO:tasks.rados.rados.0.smithi105.stdout:933: done (4 left) 2024-04-20T02:51:12.504 INFO:tasks.rados.rados.0.smithi105.stdout:934: done (3 left) 2024-04-20T02:51:12.504 INFO:tasks.rados.rados.0.smithi105.stdout:935: done (2 left) 2024-04-20T02:51:12.504 INFO:tasks.rados.rados.0.smithi105.stdout:937: done (1 left) 2024-04-20T02:51:12.504 INFO:tasks.rados.rados.0.smithi105.stdout:938: read oid 41 snap -1 2024-04-20T02:51:12.504 INFO:tasks.rados.rados.0.smithi105.stdout:938: expect (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:51:12.504 INFO:tasks.rados.rados.0.smithi105.stdout:939: write (excl) oid 50 current snap is 10 2024-04-20T02:51:12.504 INFO:tasks.rados.rados.0.smithi105.stdout:936: expect (ObjNum 135 snap 9 seq_num 135) 2024-04-20T02:51:12.507 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:12.507 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:12.525 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:12.570 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:12.570+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:12.703 INFO:tasks.rados.rados.0.smithi105.stdout:939: seq_num 146 ranges {453822=797061,1760371=653366} 2024-04-20T02:51:12.703 INFO:tasks.rados.rados.0.smithi105.stdout:939: writing smithi10548823-50 from 453822 to 1250883 tid 1 2024-04-20T02:51:12.703 INFO:tasks.rados.rados.0.smithi105.stdout:939: writing smithi10548823-50 from 1760371 to 2413737 tid 2 2024-04-20T02:51:12.703 INFO:tasks.rados.rados.0.smithi105.stdout:936: done (2 left) 2024-04-20T02:51:12.703 INFO:tasks.rados.rados.0.smithi105.stdout:940: delete oid 46 current snap is 10 2024-04-20T02:51:12.703 INFO:tasks.rados.rados.0.smithi105.stdout:938: expect (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:51:12.703 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:12.704+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:12.720 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:12.720 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:12.731 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:12.803 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:12.804+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:12.821 INFO:tasks.rados.rados.0.smithi105.stdout:939: finishing write tid 1 to smithi10548823-50 2024-04-20T02:51:12.821 INFO:tasks.rados.rados.0.smithi105.stdout:939: finishing write tid 2 to smithi10548823-50 2024-04-20T02:51:12.821 INFO:tasks.rados.rados.0.smithi105.stdout:939: finishing write tid 3 to smithi10548823-50 2024-04-20T02:51:12.821 INFO:tasks.rados.rados.0.smithi105.stdout:938: done (2 left) 2024-04-20T02:51:12.821 INFO:tasks.rados.rados.0.smithi105.stdout:940: done (1 left) 2024-04-20T02:51:12.821 INFO:tasks.rados.rados.0.smithi105.stdout:941: set_chunk oid 3 target oid 2024-04-20T02:51:12.823 INFO:tasks.rados.rados.0.smithi105.stdout:941: set_chunk oid 3 offset: 1483776 length: 1536 target oid 80422bc3d307b4a25bdafcc84ac7fb01cb55a09810e8b0f37bb12e0edb5c48ca offset: 0 snap -1 2024-04-20T02:51:12.823 INFO:tasks.rados.rados.0.smithi105.stdout:942: write (excl) oid 28 current snap is 10 2024-04-20T02:51:12.823 INFO:tasks.rados.rados.0.smithi105.stdout:939: finishing write tid 4 to smithi10548823-50 2024-04-20T02:51:12.823 INFO:tasks.rados.rados.0.smithi105.stdout:939: oid 50 updating version 0 to 38 2024-04-20T02:51:12.823 INFO:tasks.rados.rados.0.smithi105.stdout:939: oid 50 version 38 is already newer than 37 2024-04-20T02:51:12.823 INFO:tasks.rados.rados.0.smithi105.stdout:939: oid 50 version 38 is already newer than 36 2024-04-20T02:51:12.823 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 38 (ObjNum 146 snap 10 seq_num 146) dirty exists 2024-04-20T02:51:12.823 INFO:tasks.rados.rados.0.smithi105.stdout:939: left oid 50 (ObjNum 146 snap 10 seq_num 146) 2024-04-20T02:51:12.824 INFO:tasks.rados.rados.0.smithi105.stdout:942: seq_num 148 ranges {487436=547559,1760982=657035} 2024-04-20T02:51:12.829 INFO:tasks.rados.rados.0.smithi105.stdout:942: writing smithi10548823-28 from 487436 to 1034995 tid 1 2024-04-20T02:51:12.839 INFO:tasks.rados.rados.0.smithi105.stdout:942: writing smithi10548823-28 from 1760982 to 2418017 tid 2 2024-04-20T02:51:12.839 INFO:tasks.rados.rados.0.smithi105.stdout:939: done (2 left) 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:943: read oid 38 snap -1 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:943: expect deleted 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:944: read oid 30 snap -1 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:944: expect (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:945: read oid 47 snap -1 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:945: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:946: read oid 39 snap -1 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:946: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:947: tier_promote oid 45 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:948: tier_evict oid 21 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:948: tier_evict oid 21 snap -1 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:949: read oid 26 snap -1 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:949: expect (ObjNum 137 snap 9 seq_num 137) 2024-04-20T02:51:12.840 INFO:tasks.rados.rados.0.smithi105.stdout:950: set_chunk oid 1 target oid 2024-04-20T02:51:12.873 INFO:tasks.rados.rados.0.smithi105.stdout:950: set_chunk oid 1 offset: 1956352 length: 493568 target oid 10eeb6a20e0f94de6c8952d1ecd7b6512737d78615f85187d3ac125e3c85a112 offset: 0 snap 6 2024-04-20T02:51:12.873 INFO:tasks.rados.rados.0.smithi105.stdout:951: read oid 48 snap 6 2024-04-20T02:51:12.873 INFO:tasks.rados.rados.0.smithi105.stdout:951: expect (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:51:12.873 INFO:tasks.rados.rados.0.smithi105.stdout:952: read oid 11 snap 9 2024-04-20T02:51:12.873 INFO:tasks.rados.rados.0.smithi105.stdout:952: expect deleted 2024-04-20T02:51:12.873 INFO:tasks.rados.rados.0.smithi105.stdout:953: set_chunk oid 33 target oid 2024-04-20T02:51:12.885 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:12.885 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:12.896 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:953: set_chunk oid 33 offset: 48128 length: 1456640 target oid 054f0564d7bc4ba25246e28012c6a10de04c6bde6b3fb5d0189d536bd272cc88 offset: 0 snap -1 2024-04-20T02:51:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:954: read oid 31 snap -1 2024-04-20T02:51:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:954: expect (ObjNum 142 snap 9 seq_num 142) 2024-04-20T02:51:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:955: tier_flush oid 15 2024-04-20T02:51:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:955: tier_flush oid 15 snap -1 2024-04-20T02:51:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:941: finishing set_chunk to oid 3 2024-04-20T02:51:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 3 set_chunk returned error code -95 offset: 1483776 length: 1536 2024-04-20T02:51:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-20T02:51:12.906 INFO:tasks.rados.rados.0.smithi105.stdout:946: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:12.940 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:12.941+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:12.971 INFO:tasks.rados.rados.0.smithi105.stdout:948: got (22) Invalid argument 2024-04-20T02:51:12.971 INFO:tasks.rados.rados.0.smithi105.stdout:944: expect (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:51:13.006 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:13.006+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:13.025 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:13.026 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:13.036 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:13.097 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:13.142 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:13.143+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:13.206 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:13.206 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:13.207 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:13.207+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:13.216 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:13.243 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:13.246 INFO:tasks.rados.rados.0.smithi105.stdout:945: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:51:13.262 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:13.281 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:13.302 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:13.307 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:13.308+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:13.319 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:13.338 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:13.341 INFO:tasks.rados.rados.0.smithi105.stdout:949: expect (ObjNum 137 snap 9 seq_num 137) 2024-04-20T02:51:13.357 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:13.377 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:13.378 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:13.378 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:13.378 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:13.378 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:13.378 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:13.378 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:13.378 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:13.379 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:13.386 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:13.386 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:13.386 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:13.387 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:13.397 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:13.438 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:13.439+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:13.452 INFO:tasks.rados.rados.0.smithi105.stdout:942: finishing write tid 1 to smithi10548823-28 2024-04-20T02:51:13.452 INFO:tasks.rados.rados.0.smithi105.stdout:951: expect (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:51:13.461 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:51:13.462 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.5 to 1 (10000) 2024-04-20T02:51:13.475 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-20T02:51:13.475 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:51:13.509 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:13.510+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:13.569 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:13.570 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:13.570 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:13.570 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:13.570 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:13.580 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:942: finishing write tid 2 to smithi10548823-28 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:942: finishing write tid 3 to smithi10548823-28 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:941: done (14 left) 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:943: done (13 left) 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:944: done (12 left) 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:945: done (11 left) 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:946: done (10 left) 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:948: done (9 left) 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:949: done (8 left) 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:951: done (7 left) 2024-04-20T02:51:13.610 INFO:tasks.rados.rados.0.smithi105.stdout:952: done (6 left) 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:956: rollback oid 24 current snap is 10 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 24 to 6 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:942: finishing write tid 4 to smithi10548823-28 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:942: oid 28 updating version 0 to 91 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:942: oid 28 updating version 91 to 92 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:942: oid 28 updating version 92 to 93 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 93 (ObjNum 148 snap 10 seq_num 148) dirty exists 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:942: left oid 28 (ObjNum 148 snap 10 seq_num 148) 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:950: finishing set_chunk to oid 1 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:947: got (0) Success 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:955: got (2) No such file or directory 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:955: got expected ENOENT (src dne) 2024-04-20T02:51:13.611 INFO:tasks.rados.rados.0.smithi105.stdout:942: done (6 left) 2024-04-20T02:51:13.612 INFO:tasks.rados.rados.0.smithi105.stdout:947: done (5 left) 2024-04-20T02:51:13.612 INFO:tasks.rados.rados.0.smithi105.stdout:950: done (4 left) 2024-04-20T02:51:13.612 INFO:tasks.rados.rados.0.smithi105.stdout:955: done (3 left) 2024-04-20T02:51:13.612 INFO:tasks.rados.rados.0.smithi105.stdout:957: delete oid 34 current snap is 10 2024-04-20T02:51:13.612 INFO:tasks.rados.rados.0.smithi105.stdout:954: expect (ObjNum 142 snap 9 seq_num 142) 2024-04-20T02:51:13.621 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:13.622+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:13.714 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:13.714+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:13.744 INFO:tasks.rados.rados.0.smithi105.stdout:953: finishing set_chunk to oid 33 2024-04-20T02:51:13.744 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 25 (ObjNum 129 snap 9 seq_num 129) dirty exists 2024-04-20T02:51:13.744 INFO:tasks.rados.rados.0.smithi105.stdout:953: done (3 left) 2024-04-20T02:51:13.744 INFO:tasks.rados.rados.0.smithi105.stdout:954: done (2 left) 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:957: done (1 left) 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:958: read oid 17 snap -1 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:958: expect (ObjNum 123 snap 7 seq_num 123) 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:959: tier_promote oid 42 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:956: finishing rollback tid 0 to smithi10548823-24 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:956: finishing rollback tid 1 to smithi10548823-24 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:956: finishing rollback tid 2 to smithi10548823-24 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 138 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:956: done (2 left) 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:960: read oid 43 snap 9 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:960: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:961: write (excl) oid 46 current snap is 10 2024-04-20T02:51:13.745 INFO:tasks.rados.rados.0.smithi105.stdout:961: seq_num 150 ranges {521050=698057,1877946=643104} 2024-04-20T02:51:13.751 INFO:tasks.rados.rados.0.smithi105.stdout:961: writing smithi10548823-46 from 521050 to 1219107 tid 1 2024-04-20T02:51:13.753 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:13.753 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:13.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:13.753 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:13.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.753 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:13.753 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:13.753 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.015509109, 2024-04-20T02:51:13.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:51:13.754 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.754 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.754 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:13.754 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:13.754 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:13.754 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:51:13.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.754 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:13.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:51:13.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:51:13.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:51:13.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:51:13.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:51:13.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:51:13.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:13.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:51:13.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:51:13.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.761 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:13.761 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.984220048000001, 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:13.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:51:13.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:51:13.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:51:13.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:51:13.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:51:13.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:51:13.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:51:13.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:51:13.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:51:13.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:51:13.768 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.768 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.768 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.769 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:13.769 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.979529591999999, 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:13.775 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.972773123, 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:51:13.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:51:13.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:51:13.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:51:13.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.962957612, 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.781 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.782 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:13.782 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:13.782 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:13.782 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:51:13.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.782 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:13.787 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.961913679999999, 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:51:13.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:51:13.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:51:13.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.793 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.961892519999999, 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:51:13.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:13.798 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.754321939, 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:13.802 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.740127294000001, 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:51:13.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:51:13.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.806 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.734160036999999, 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:51:13.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:51:13.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:51:13.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.725754034000001, 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.622121993, 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.391525683000001, 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.819 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:13.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:51:13.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:51:13.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.364581533999999, 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:51:13.823 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:13.827 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.339223267000001, 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:51:13.831 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.311082823, 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.273714217999999, 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:51:13.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.273417295000002, 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.840 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:51:13.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:51:13.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.268334177, 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:51:13.844 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:51:13.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:51:13.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:51:13.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:51:13.848 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.848 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.848 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.241331618, 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:13.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:13.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:51:13.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:51:13.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:13.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:51:13.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:13.854 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:13.858 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:13.858 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:13.856+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:13.859 INFO:tasks.rados.rados.0.smithi105.stdout:961: writing smithi10548823-46 from 1877946 to 2521050 tid 2 2024-04-20T02:51:13.859 INFO:tasks.rados.rados.0.smithi105.stdout:962: write (excl) oid 13 current snap is 10 2024-04-20T02:51:13.859 INFO:tasks.rados.rados.0.smithi105.stdout:962: seq_num 151 ranges {537857=773306,1936428=520715} 2024-04-20T02:51:13.859 INFO:tasks.rados.rados.0.smithi105.stdout:962: writing smithi10548823-13 from 537857 to 1311163 tid 1 2024-04-20T02:51:13.859 INFO:tasks.rados.rados.0.smithi105.stdout:962: writing smithi10548823-13 from 1936428 to 2457143 tid 2 2024-04-20T02:51:13.859 INFO:tasks.rados.rados.0.smithi105.stdout:963: set_chunk oid 18 target oid 2024-04-20T02:51:13.859 INFO:tasks.rados.rados.0.smithi105.stdout:963: set_chunk oid 18 offset: 2075136 length: 904704 target oid 801c16b410980046d7785aaebfd4d3724ad7094d4b7e7d42d7e1b3cde627c92d offset: 0 snap -1 2024-04-20T02:51:13.859 INFO:tasks.rados.rados.0.smithi105.stdout:964: write (excl) oid 49 current snap is 10 2024-04-20T02:51:13.859 INFO:tasks.rados.rados.0.smithi105.stdout:964: seq_num 152 ranges {554664=448555,1478557=664373} 2024-04-20T02:51:13.860 INFO:tasks.rados.rados.0.smithi105.stdout:964: writing smithi10548823-49 from 554664 to 1003219 tid 1 2024-04-20T02:51:13.860 INFO:tasks.rados.rados.0.smithi105.stdout:964: writing smithi10548823-49 from 1478557 to 2142930 tid 2 2024-04-20T02:51:13.860 INFO:tasks.rados.rados.0.smithi105.stdout:965: delete oid 45 current snap is 10 2024-04-20T02:51:13.860 INFO:tasks.rados.rados.0.smithi105.stdout:959: got (0) Success 2024-04-20T02:51:13.860 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 21 (ObjNum 108 snap 6 seq_num 108) dirty exists 2024-04-20T02:51:13.860 INFO:tasks.rados.rados.0.smithi105.stdout:960: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:51:13.862 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:51:13.863 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:51:13.864 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-04-20T02:51:13.456632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":278233030,"num_objects":211,"num_object_clones":35,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":4,"num_read":1669,"num_read_kb":1266632,"num_write":1308,"num_write_kb":350011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":45431190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":54,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":87,"acting":87,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":697604,"kb_used_data":469776,"kb_used_omap":12,"kb_used_meta":224947,"kb_avail":754277116,"statfs":{"total":773094113280,"available":772379766784,"internally_reserved":0,"allocated":481050624,"data_stored":555194509,"data_compressed":12803180,"data_compressed_allocated":13479936,"data_compressed_original":31988855,"omap_allocated":12807,"internal_metadata":230346233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7907732,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":191,"num_read_kb":261919,"num_write":50,"num_write_kb":18764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008356"},"pg_stats":[{"pgid":"4.f","version":"34'15","reported_seq":98,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.449050+0000","last_change":"2024-04-20T02:51:12.449411+0000","last_active":"2024-04-20T02:51:12.449050+0000","last_peered":"2024-04-20T02:51:12.449050+0000","last_clean":"2024-04-20T02:51:12.449050+0000","last_became_active":"2024-04-20T02:50:35.245323+0000","last_became_peered":"2024-04-20T02:50:35.245323+0000","last_unstale":"2024-04-20T02:51:12.449050+0000","last_undegraded":"2024-04-20T02:51:12.449050+0000","last_fullsized":"2024-04-20T02:51:12.449050+0000","mapping_epoch":22,"log_start":"33'10","ondisk_log_start":"33'10","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":10,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:53.742736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99966113099999998,"stat_sum":{"num_bytes":3428595,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":8320,"num_write":14,"num_write_kb":4030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.e","version":"51'30","reported_seq":108,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.898590+0000","last_change":"2024-04-20T02:51:06.133569+0000","last_active":"2024-04-20T02:51:08.898590+0000","last_peered":"2024-04-20T02:51:08.898590+0000","last_clean":"2024-04-20T02:51:08.898590+0000","last_became_active":"2024-04-20T02:50:35.244872+0000","last_became_peered":"2024-04-20T02:50:35.244872+0000","last_unstale":"2024-04-20T02:51:08.898590+0000","last_undegraded":"2024-04-20T02:51:08.898590+0000","last_fullsized":"2024-04-20T02:51:08.898590+0000","mapping_epoch":22,"log_start":"49'27","ondisk_log_start":"49'27","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":27,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:39.148211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014166396,"stat_sum":{"num_bytes":5761953,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":26678,"num_write":20,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.d","version":"51'64","reported_seq":180,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.914763+0000","last_change":"2024-04-20T02:51:06.140431+0000","last_active":"2024-04-20T02:51:08.914763+0000","last_peered":"2024-04-20T02:51:08.914763+0000","last_clean":"2024-04-20T02:51:08.914763+0000","last_became_active":"2024-04-20T02:50:35.528107+0000","last_became_peered":"2024-04-20T02:50:35.528107+0000","last_unstale":"2024-04-20T02:51:08.914763+0000","last_undegraded":"2024-04-20T02:51:08.914763+0000","last_fullsized":"2024-04-20T02:51:08.914763+0000","mapping_epoch":22,"log_start":"49'61","ondisk_log_start":"49'61","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:59.747989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020978734999999998,"stat_sum":{"num_bytes":16925968,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":138,"num_read_kb":157400,"num_write":47,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.c","version":"49'39","reported_seq":118,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.840601+0000","last_change":"2024-04-20T02:51:06.128048+0000","last_active":"2024-04-20T02:51:08.840601+0000","last_peered":"2024-04-20T02:51:08.840601+0000","last_clean":"2024-04-20T02:51:08.840601+0000","last_became_active":"2024-04-20T02:50:35.241480+0000","last_became_peered":"2024-04-20T02:50:35.241480+0000","last_unstale":"2024-04-20T02:51:08.840601+0000","last_undegraded":"2024-04-20T02:51:08.840601+0000","last_fullsized":"2024-04-20T02:51:08.840601+0000","mapping_epoch":22,"log_start":"42'33","ondisk_log_start":"42'33","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":33,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.915698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087151550000000005,"stat_sum":{"num_bytes":7683485,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":31347,"num_write":31,"num_write_kb":8780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.b","version":"52'71","reported_seq":175,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.518160+0000","last_change":"2024-04-20T02:51:06.126734+0000","last_active":"2024-04-20T02:51:09.518160+0000","last_peered":"2024-04-20T02:51:09.518160+0000","last_clean":"2024-04-20T02:51:09.518160+0000","last_became_active":"2024-04-20T02:50:35.242331+0000","last_became_peered":"2024-04-20T02:50:35.242331+0000","last_unstale":"2024-04-20T02:51:09.518160+0000","last_undegraded":"2024-04-20T02:51:09.518160+0000","last_fullsized":"2024-04-20T02:51:09.518160+0000","mapping_epoch":22,"log_start":"48'68","ondisk_log_start":"48'68","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.773051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064970080000000003,"stat_sum":{"num_bytes":14033713,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":94,"num_read_kb":102342,"num_write":56,"num_write_kb":15626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.a","version":"53'33","reported_seq":147,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.469786+0000","last_change":"2024-04-20T02:51:12.470019+0000","last_active":"2024-04-20T02:51:12.469786+0000","last_peered":"2024-04-20T02:51:12.469786+0000","last_clean":"2024-04-20T02:51:12.469786+0000","last_became_active":"2024-04-20T02:50:35.413503+0000","last_became_peered":"2024-04-20T02:50:35.413503+0000","last_unstale":"2024-04-20T02:51:12.469786+0000","last_undegraded":"2024-04-20T02:51:12.469786+0000","last_fullsized":"2024-04-20T02:51:12.469786+0000","mapping_epoch":22,"log_start":"44'29","ondisk_log_start":"44'29","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":29,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:57.185462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014460114,"stat_sum":{"num_bytes":8235364,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":99,"num_read_kb":115384,"num_write":28,"num_write_kb":7354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.9","version":"50'114","reported_seq":248,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.840448+0000","last_change":"2024-04-20T02:51:06.127431+0000","last_active":"2024-04-20T02:51:08.840448+0000","last_peered":"2024-04-20T02:51:08.840448+0000","last_clean":"2024-04-20T02:51:08.840448+0000","last_became_active":"2024-04-20T02:50:35.244955+0000","last_became_peered":"2024-04-20T02:50:35.244955+0000","last_unstale":"2024-04-20T02:51:08.840448+0000","last_undegraded":"2024-04-20T02:51:08.840448+0000","last_fullsized":"2024-04-20T02:51:08.840448+0000","mapping_epoch":22,"log_start":"50'110","ondisk_log_start":"50'110","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":110,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:51.105656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093707679999999998,"stat_sum":{"num_bytes":26183771,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":179,"num_read_kb":183049,"num_write":88,"num_write_kb":25797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.8","version":"48'47","reported_seq":136,"reported_epoch":51,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:08.840468+0000","last_change":"2024-04-20T02:51:02.924920+0000","last_active":"2024-04-20T02:51:08.840468+0000","last_peered":"2024-04-20T02:51:08.840468+0000","last_clean":"2024-04-20T02:50:58.916741+0000","last_became_active":"2024-04-20T02:51:02.922909+0000","last_became_peered":"2024-04-20T02:51:02.922909+0000","last_unstale":"2024-04-20T02:51:08.840468+0000","last_undegraded":"2024-04-20T02:51:08.840468+0000","last_fullsized":"2024-04-20T02:51:08.840468+0000","mapping_epoch":44,"log_start":"42'44","ondisk_log_start":"42'44","created":22,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":44,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.192506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041300199999999999,"stat_sum":{"num_bytes":10144194,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":54887,"num_write":36,"num_write_kb":8549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[4,7,6],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-20T02:51:10.445505+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:51:10.445505+0000","last_peered":"2024-04-20T02:51:10.445505+0000","last_clean":"2024-04-20T02:51:10.445505+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:51:10.445505+0000","last_undegraded":"2024-04-20T02:51:10.445505+0000","last_fullsized":"2024-04-20T02:51:10.445505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":94,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.449497+0000","last_change":"2024-04-20T02:51:12.449685+0000","last_active":"2024-04-20T02:51:12.449497+0000","last_peered":"2024-04-20T02:51:12.449497+0000","last_clean":"2024-04-20T02:51:12.449497+0000","last_became_active":"2024-04-20T02:51:01.129374+0000","last_became_peered":"2024-04-20T02:51:01.129374+0000","last_unstale":"2024-04-20T02:51:12.449497+0000","last_undegraded":"2024-04-20T02:51:12.449497+0000","last_fullsized":"2024-04-20T02:51:12.449497+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:48.451347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99990455700000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.4","version":"54'16","reported_seq":105,"reported_epoch":54,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:12.449764+0000","last_change":"2024-04-20T02:51:12.449764+0000","last_active":"2024-04-20T02:51:12.449764+0000","last_peered":"2024-04-20T02:51:12.449764+0000","last_clean":"2024-04-20T02:51:12.449764+0000","last_became_active":"2024-04-20T02:50:35.528215+0000","last_became_peered":"2024-04-20T02:50:35.528215+0000","last_unstale":"2024-04-20T02:51:12.449764+0000","last_undegraded":"2024-04-20T02:51:12.449764+0000","last_fullsized":"2024-04-20T02:51:12.449764+0000","mapping_epoch":22,"log_start":"49'12","ondisk_log_start":"49'12","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":12,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99860969499999996,"stat_sum":{"num_bytes":3092455,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":53444,"num_write":10,"num_write_kb":2385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.459377+0000","last_change":"2024-04-20T02:51:01.134074+0000","last_active":"2024-04-20T02:51:11.459377+0000","last_peered":"2024-04-20T02:51:11.459377+0000","last_clean":"2024-04-20T02:51:11.459377+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:51:11.459377+0000","last_undegraded":"2024-04-20T02:51:11.459377+0000","last_fullsized":"2024-04-20T02:51:11.459377+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:38.091564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"52'89","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.543472+0000","last_change":"2024-04-20T02:50:34.242061+0000","last_active":"2024-04-20T02:51:09.543472+0000","last_peered":"2024-04-20T02:51:09.543472+0000","last_clean":"2024-04-20T02:51:09.543472+0000","last_became_active":"2024-04-20T02:50:34.241638+0000","last_became_peered":"2024-04-20T02:50:34.241638+0000","last_unstale":"2024-04-20T02:51:09.543472+0000","last_undegraded":"2024-04-20T02:51:09.543472+0000","last_fullsized":"2024-04-20T02:51:09.543472+0000","mapping_epoch":21,"log_start":"46'86","ondisk_log_start":"46'86","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":92,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:58.050023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27403590,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":82,"num_read_kb":3703,"num_write":114,"num_write_kb":34796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"53'38","reported_seq":174,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.451318+0000","last_change":"2024-04-20T02:51:12.451512+0000","last_active":"2024-04-20T02:51:12.451318+0000","last_peered":"2024-04-20T02:51:12.451318+0000","last_clean":"2024-04-20T02:51:12.451318+0000","last_became_active":"2024-04-20T02:51:04.417108+0000","last_became_peered":"2024-04-20T02:51:04.417108+0000","last_unstale":"2024-04-20T02:51:12.451318+0000","last_undegraded":"2024-04-20T02:51:12.451318+0000","last_fullsized":"2024-04-20T02:51:12.451318+0000","mapping_epoch":46,"log_start":"42'33","ondisk_log_start":"42'33","created":22,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":33,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:00.545359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.000954766,"stat_sum":{"num_bytes":7640560,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":54652,"num_write":33,"num_write_kb":9882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5515811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.464845+0000","last_change":"2024-04-20T02:50:28.091742+0000","last_active":"2024-04-20T02:51:11.464845+0000","last_peered":"2024-04-20T02:51:11.464845+0000","last_clean":"2024-04-20T02:51:11.464845+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:51:11.464845+0000","last_undegraded":"2024-04-20T02:51:11.464845+0000","last_fullsized":"2024-04-20T02:51:11.464845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:32.326186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"51'93","reported_seq":142,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-20T02:51:10.445074+0000","last_change":"2024-04-20T02:50:34.241909+0000","last_active":"2024-04-20T02:51:10.445074+0000","last_peered":"2024-04-20T02:51:10.445074+0000","last_clean":"2024-04-20T02:51:10.445074+0000","last_became_active":"2024-04-20T02:50:34.241613+0000","last_became_peered":"2024-04-20T02:50:34.241613+0000","last_unstale":"2024-04-20T02:51:10.445074+0000","last_undegraded":"2024-04-20T02:51:10.445074+0000","last_fullsized":"2024-04-20T02:51:10.445074+0000","mapping_epoch":21,"log_start":"45'90","ondisk_log_start":"45'90","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":102,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:54.710965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19924935,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":82,"num_read_kb":1868,"num_write":122,"num_write_kb":36775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'32","reported_seq":135,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.469721+0000","last_change":"2024-04-20T02:51:12.469951+0000","last_active":"2024-04-20T02:51:12.469721+0000","last_peered":"2024-04-20T02:51:12.469721+0000","last_clean":"2024-04-20T02:51:12.469721+0000","last_became_active":"2024-04-20T02:50:35.413590+0000","last_became_peered":"2024-04-20T02:50:35.413590+0000","last_unstale":"2024-04-20T02:51:12.469721+0000","last_undegraded":"2024-04-20T02:51:12.469721+0000","last_fullsized":"2024-04-20T02:51:12.469721+0000","mapping_epoch":22,"log_start":"42'28","ondisk_log_start":"42'28","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":28,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:44.785443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014352482999999999,"stat_sum":{"num_bytes":9479991,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":88948,"num_write":23,"num_write_kb":7717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.459300+0000","last_change":"2024-04-20T02:51:01.151894+0000","last_active":"2024-04-20T02:51:11.459300+0000","last_peered":"2024-04-20T02:51:11.459300+0000","last_clean":"2024-04-20T02:51:11.459300+0000","last_became_active":"2024-04-20T02:51:01.151544+0000","last_became_peered":"2024-04-20T02:51:01.151544+0000","last_unstale":"2024-04-20T02:51:11.459300+0000","last_undegraded":"2024-04-20T02:51:11.459300+0000","last_fullsized":"2024-04-20T02:51:11.459300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:29.245779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.899914+0000","last_change":"2024-04-20T02:50:19.923431+0000","last_active":"2024-04-20T02:51:08.899914+0000","last_peered":"2024-04-20T02:51:08.899914+0000","last_clean":"2024-04-20T02:51:08.899914+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:51:08.899914+0000","last_undegraded":"2024-04-20T02:51:08.899914+0000","last_fullsized":"2024-04-20T02:51:08.899914+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:50:17.569030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.655607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"53'105","reported_seq":245,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.447735+0000","last_change":"2024-04-20T02:51:07.118974+0000","last_active":"2024-04-20T02:51:11.447735+0000","last_peered":"2024-04-20T02:51:11.447735+0000","last_clean":"2024-04-20T02:51:11.447735+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:51:11.447735+0000","last_undegraded":"2024-04-20T02:51:11.447735+0000","last_fullsized":"2024-04-20T02:51:11.447735+0000","mapping_epoch":49,"log_start":"52'102","ondisk_log_start":"52'102","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":110,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:42.672805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27256345,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":92,"num_read_kb":5490,"num_write":122,"num_write_kb":50638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"52'43","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.470545+0000","last_change":"2024-04-20T02:51:06.126486+0000","last_active":"2024-04-20T02:51:09.470545+0000","last_peered":"2024-04-20T02:51:09.470545+0000","last_clean":"2024-04-20T02:51:09.470545+0000","last_became_active":"2024-04-20T02:50:35.528177+0000","last_became_peered":"2024-04-20T02:50:35.528177+0000","last_unstale":"2024-04-20T02:51:09.470545+0000","last_undegraded":"2024-04-20T02:51:09.470545+0000","last_fullsized":"2024-04-20T02:51:09.470545+0000","mapping_epoch":22,"log_start":"49'39","ondisk_log_start":"49'39","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":39,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:55.042511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065151180000000003,"stat_sum":{"num_bytes":12790883,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":78,"num_read_kb":73062,"num_write":53,"num_write_kb":13339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.899929+0000","last_change":"2024-04-20T02:50:28.092115+0000","last_active":"2024-04-20T02:51:08.899929+0000","last_peered":"2024-04-20T02:51:08.899929+0000","last_clean":"2024-04-20T02:51:08.899929+0000","last_became_active":"2024-04-20T02:50:28.091570+0000","last_became_peered":"2024-04-20T02:50:28.091570+0000","last_unstale":"2024-04-20T02:51:08.899929+0000","last_undegraded":"2024-04-20T02:51:08.899929+0000","last_fullsized":"2024-04-20T02:51:08.899929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:27.403210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"48'116","reported_seq":174,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.792164+0000","last_change":"2024-04-20T02:50:34.239216+0000","last_active":"2024-04-20T02:51:08.792164+0000","last_peered":"2024-04-20T02:51:08.792164+0000","last_clean":"2024-04-20T02:51:08.792164+0000","last_became_active":"2024-04-20T02:50:34.238906+0000","last_became_peered":"2024-04-20T02:50:34.238906+0000","last_unstale":"2024-04-20T02:51:08.792164+0000","last_undegraded":"2024-04-20T02:51:08.792164+0000","last_fullsized":"2024-04-20T02:51:08.792164+0000","mapping_epoch":21,"log_start":"42'113","ondisk_log_start":"42'113","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":138,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.390119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28959845,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":111,"num_read_kb":6350,"num_write":143,"num_write_kb":48210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"52'127","reported_seq":297,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.436361+0000","last_change":"2024-04-20T02:51:06.140804+0000","last_active":"2024-04-20T02:51:09.436361+0000","last_peered":"2024-04-20T02:51:09.436361+0000","last_clean":"2024-04-20T02:51:09.436361+0000","last_became_active":"2024-04-20T02:51:05.421483+0000","last_became_peered":"2024-04-20T02:51:05.421483+0000","last_unstale":"2024-04-20T02:51:09.436361+0000","last_undegraded":"2024-04-20T02:51:09.436361+0000","last_fullsized":"2024-04-20T02:51:09.436361+0000","mapping_epoch":47,"log_start":"49'124","ondisk_log_start":"49'124","created":22,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.981337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020740003999999999,"stat_sum":{"num_bytes":23383373,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":139,"num_read_kb":154428,"num_write":95,"num_write_kb":24765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":18280879,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.898181+0000","last_change":"2024-04-20T02:50:28.091938+0000","last_active":"2024-04-20T02:51:08.898181+0000","last_peered":"2024-04-20T02:51:08.898181+0000","last_clean":"2024-04-20T02:51:08.898181+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:51:08.898181+0000","last_undegraded":"2024-04-20T02:51:08.898181+0000","last_fullsized":"2024-04-20T02:51:08.898181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:34.990718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-20T02:51:10.445522+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:51:10.445522+0000","last_peered":"2024-04-20T02:51:10.445522+0000","last_clean":"2024-04-20T02:51:10.445522+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:51:10.445522+0000","last_undegraded":"2024-04-20T02:51:10.445522+0000","last_fullsized":"2024-04-20T02:51:10.445522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"54'67","reported_seq":185,"reported_epoch":54,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:12.452284+0000","last_change":"2024-04-20T02:51:12.450055+0000","last_active":"2024-04-20T02:51:12.452284+0000","last_peered":"2024-04-20T02:51:12.452284+0000","last_clean":"2024-04-20T02:51:12.452284+0000","last_became_active":"2024-04-20T02:50:35.413445+0000","last_became_peered":"2024-04-20T02:50:35.413445+0000","last_unstale":"2024-04-20T02:51:12.452284+0000","last_undegraded":"2024-04-20T02:51:12.452284+0000","last_fullsized":"2024-04-20T02:51:12.452284+0000","mapping_epoch":22,"log_start":"51'61","ondisk_log_start":"51'61","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":61,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:56.404298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0023455880000001,"stat_sum":{"num_bytes":10840416,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":105,"num_read_kb":85234,"num_write":44,"num_write_kb":11536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.450251+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:51:11.450251+0000","last_peered":"2024-04-20T02:51:11.450251+0000","last_clean":"2024-04-20T02:51:11.450251+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:51:11.450251+0000","last_undegraded":"2024-04-20T02:51:11.450251+0000","last_fullsized":"2024-04-20T02:51:11.450251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.432280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"52'79","reported_seq":218,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.549410+0000","last_change":"2024-04-20T02:51:06.119828+0000","last_active":"2024-04-20T02:51:09.549410+0000","last_peered":"2024-04-20T02:51:09.549410+0000","last_clean":"2024-04-20T02:51:09.549410+0000","last_became_active":"2024-04-20T02:51:04.413969+0000","last_became_peered":"2024-04-20T02:51:04.413969+0000","last_unstale":"2024-04-20T02:51:09.549410+0000","last_undegraded":"2024-04-20T02:51:09.549410+0000","last_fullsized":"2024-04-20T02:51:09.549410+0000","mapping_epoch":46,"log_start":"52'76","ondisk_log_start":"52'76","created":22,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":76,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.956090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071174699999999999,"stat_sum":{"num_bytes":14473226,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":70,"num_read_kb":59946,"num_write":70,"num_write_kb":18690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7407305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":174097947,"num_objects":82,"num_object_clones":35,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":4,"num_read":1186,"num_read_kb":1249121,"num_write":648,"num_write_kb":177176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":31203995,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":54,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":318308352,"data_stored":381035946,"data_compressed":1651560,"data_compressed_allocated":1400832,"data_compressed_original":4062327,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":48,"acting":48,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":103544715,"num_objects":127,"num_object_clones":0,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":367,"num_read_kb":17411,"num_write":501,"num_write_kb":170419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":161460224,"data_stored":175351286,"data_compressed":11184321,"data_compressed_allocated":11980800,"data_compressed_original":26456064,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509456,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67260,"kb_used_data":40168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94304580,"statfs":{"total":96636764160,"available":96567889920,"internally_reserved":0,"allocated":41132032,"data_stored":52309410,"data_compressed":16844,"data_compressed_allocated":45056,"data_compressed_original":577536,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757253,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75860,"kb_used_data":45520,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94295980,"statfs":{"total":96636764160,"available":96559083520,"internally_reserved":0,"allocated":46612480,"data_stored":52416628,"data_compressed":1083307,"data_compressed_allocated":1159168,"data_compressed_original":2473984,"omap_allocated":1657,"internal_metadata":28375431},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56988,"kb_used_data":29968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94314852,"statfs":{"total":96636764160,"available":96578408448,"internally_reserved":0,"allocated":30687232,"data_stored":36085808,"data_compressed":1041091,"data_compressed_allocated":1048576,"data_compressed_original":2203648,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91444,"kb_used_data":64424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94280396,"statfs":{"total":96636764160,"available":96543125504,"internally_reserved":0,"allocated":65970176,"data_stored":75138624,"data_compressed":1361937,"data_compressed_allocated":1327104,"data_compressed_original":3099005,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":54088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":55386112,"data_stored":63810496,"data_compressed":2040256,"data_compressed_allocated":2162688,"data_compressed_original":4795990,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.217}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98756,"kb_used_data":71728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94273084,"statfs":{"total":96636764160,"available":96535638016,"internally_reserved":0,"allocated":73449472,"data_stored":80310869,"data_compressed":3034714,"data_compressed_allocated":3219456,"data_compressed_original":7592743,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509452,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101592,"kb_used_data":70400,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94270248,"statfs":{"total":96636764160,"available":96532733952,"internally_reserved":0,"allocated":72089600,"data_stored":84378489,"data_compressed":2095144,"data_compressed_allocated":2289664,"data_compressed_original":5824512,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509454,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":124596,"kb_used_data":93480,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94247244,"statfs":{"total":96636764160,"available":96509177856,"internally_reserved":0,"allocated":95723520,"data_stored":110744185,"data_compressed":2129887,"data_compressed_allocated":2228224,"data_compressed_original":5421437,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28381184,"data_stored":30949639,"data_compressed":1843809,"data_compressed_allocated":1990656,"data_compressed_original":4657152,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11038720,"data_stored":11916936,"data_compressed":822943,"data_compressed_allocated":872448,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11038720,"data_stored":11916936,"data_compressed":822943,"data_compressed_allocated":872448,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28123136,"data_stored":30409704,"data_compressed":1915052,"data_compressed_allocated":2048000,"data_compressed_original":4419584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28381184,"data_stored":30949639,"data_compressed":1843809,"data_compressed_allocated":1990656,"data_compressed_original":4657152,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14135296,"data_stored":15322309,"data_compressed":1048539,"data_compressed_allocated":1118208,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40361984,"data_stored":43886123,"data_compressed":2887226,"data_compressed_allocated":3088384,"data_compressed_original":6758400,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44679168,"data_stored":55212658,"data_compressed":294672,"data_compressed_allocated":266240,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54341632,"data_stored":62977902,"data_compressed":538994,"data_compressed_allocated":454656,"data_compressed_original":1288573,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19058688,"data_stored":23925086,"data_compressed":218148,"data_compressed_allocated":176128,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26673152,"data_stored":33157006,"data_compressed":125204,"data_compressed_allocated":114688,"data_compressed_original":376406,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":68710400,"data_stored":81713216,"data_compressed":286078,"data_compressed_allocated":237568,"data_compressed_original":764285,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":31875072,"data_stored":36844968,"data_compressed":34768,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32444416,"data_stored":35652032,"data_compressed":142170,"data_compressed_allocated":98304,"data_compressed_original":322343,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40525824,"data_stored":51553078,"data_compressed":11526,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:51:13.866 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:51:13.909 INFO:tasks.rados.rados.0.smithi105.stdout:958: expect (ObjNum 123 snap 7 seq_num 123) 2024-04-20T02:51:13.957 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:13.957+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:14.026 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:14.026 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:14.027 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:14.027 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:14.031 INFO:tasks.rados.rados.0.smithi105.stdout:958: done (7 left) 2024-04-20T02:51:14.031 INFO:tasks.rados.rados.0.smithi105.stdout:959: done (6 left) 2024-04-20T02:51:14.031 INFO:tasks.rados.rados.0.smithi105.stdout:960: done (5 left) 2024-04-20T02:51:14.031 INFO:tasks.rados.rados.0.smithi105.stdout:965: done (4 left) 2024-04-20T02:51:14.031 INFO:tasks.rados.rados.0.smithi105.stdout:966: read oid 5 snap -1 2024-04-20T02:51:14.031 INFO:tasks.rados.rados.0.smithi105.stdout:966: expect (ObjNum 144 snap 9 seq_num 144) 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:961: finishing write tid 1 to smithi10548823-46 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:961: finishing write tid 2 to smithi10548823-46 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:961: finishing write tid 3 to smithi10548823-46 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:961: finishing write tid 4 to smithi10548823-46 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:961: oid 46 updating version 0 to 42 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:961: oid 46 version 42 is already newer than 41 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:961: oid 46 updating version 42 to 43 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 43 (ObjNum 150 snap 10 seq_num 150) dirty exists 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:961: left oid 46 (ObjNum 150 snap 10 seq_num 150) 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:963: finishing set_chunk to oid 18 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 26 (ObjNum 83 snap 1 seq_num 83) dirty exists 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:962: finishing write tid 1 to smithi10548823-13 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:962: finishing write tid 2 to smithi10548823-13 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:962: finishing write tid 3 to smithi10548823-13 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:962: finishing write tid 4 to smithi10548823-13 2024-04-20T02:51:14.032 INFO:tasks.rados.rados.0.smithi105.stdout:962: oid 13 updating version 0 to 140 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:962: oid 13 updating version 140 to 141 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:962: oid 13 updating version 141 to 142 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 142 (ObjNum 151 snap 10 seq_num 151) dirty exists 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:962: left oid 13 (ObjNum 151 snap 10 seq_num 151) 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:964: finishing write tid 1 to smithi10548823-49 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:964: finishing write tid 2 to smithi10548823-49 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:964: finishing write tid 3 to smithi10548823-49 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:964: finishing write tid 4 to smithi10548823-49 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:964: oid 49 updating version 0 to 82 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:964: oid 49 version 82 is already newer than 80 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:964: oid 49 version 82 is already newer than 81 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 82 (ObjNum 152 snap 10 seq_num 152) dirty exists 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:964: left oid 49 (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:961: done (4 left) 2024-04-20T02:51:14.033 INFO:tasks.rados.rados.0.smithi105.stdout:962: done (3 left) 2024-04-20T02:51:14.034 INFO:tasks.rados.rados.0.smithi105.stdout:963: done (2 left) 2024-04-20T02:51:14.034 INFO:tasks.rados.rados.0.smithi105.stdout:964: done (1 left) 2024-04-20T02:51:14.034 INFO:tasks.rados.rados.0.smithi105.stdout:967: tier_flush oid 41 2024-04-20T02:51:14.034 INFO:tasks.rados.rados.0.smithi105.stdout:967: tier_flush oid 41 snap -1 2024-04-20T02:51:14.034 INFO:tasks.rados.rados.0.smithi105.stdout:968: tier_evict oid 30 2024-04-20T02:51:14.034 INFO:tasks.rados.rados.0.smithi105.stdout:968: tier_evict oid 30 snap -1 2024-04-20T02:51:14.034 INFO:tasks.rados.rados.0.smithi105.stdout:969: set_chunk oid 12 target oid 2024-04-20T02:51:14.038 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:14.065 INFO:tasks.rados.rados.0.smithi105.stdout:969: set_chunk oid 12 offset: 170496 length: 1396224 target oid 0091dbb1bc5d71332801a442aa2d2bc1051691947f0220a843f3b9f50b4964e5 offset: 0 snap -1 2024-04-20T02:51:14.065 INFO:tasks.rados.rados.0.smithi105.stdout:970: write (excl) oid 48 current snap is 10 2024-04-20T02:51:14.065 INFO:tasks.rados.rados.0.smithi105.stdout:968: got (22) Invalid argument 2024-04-20T02:51:14.065 INFO:tasks.rados.rados.0.smithi105.stdout:970: seq_num 154 ranges {588278=715406,1711874=668042} 2024-04-20T02:51:14.072 INFO:tasks.rados.rados.0.smithi105.stdout:970: writing smithi10548823-48 from 588278 to 1303684 tid 1 2024-04-20T02:51:14.078 INFO:tasks.rados.rados.0.smithi105.stdout:970: writing smithi10548823-48 from 1711874 to 2379916 tid 2 2024-04-20T02:51:14.078 INFO:tasks.rados.rados.0.smithi105.stdout:968: done (4 left) 2024-04-20T02:51:14.078 INFO:tasks.rados.rados.0.smithi105.stdout:971: read oid 6 snap -1 2024-04-20T02:51:14.079 INFO:tasks.rados.rados.0.smithi105.stdout:971: expect deleted 2024-04-20T02:51:14.079 INFO:tasks.rados.rados.0.smithi105.stdout:972: write oid 2 current snap is 10 2024-04-20T02:51:14.079 INFO:tasks.rados.rados.0.smithi105.stdout:972: seq_num 155 ranges {605085=790655,2170356=434729} 2024-04-20T02:51:14.079 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:14.079+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:14.085 INFO:tasks.rados.rados.0.smithi105.stdout:972: writing smithi10548823-2 from 605085 to 1395740 tid 1 2024-04-20T02:51:14.094 INFO:tasks.rados.rados.0.smithi105.stdout:972: writing smithi10548823-2 from 2170356 to 2605085 tid 2 2024-04-20T02:51:14.094 INFO:tasks.rados.rados.0.smithi105.stdout:973: read oid 4 snap -1 2024-04-20T02:51:14.094 INFO:tasks.rados.rados.0.smithi105.stdout:973: expect (ObjNum 96 snap 4 seq_num 96) 2024-04-20T02:51:14.094 INFO:tasks.rados.rados.0.smithi105.stdout:974: read oid 49 snap -1 2024-04-20T02:51:14.094 INFO:tasks.rados.rados.0.smithi105.stdout:974: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:14.094 INFO:tasks.rados.rados.0.smithi105.stdout:975: read oid 30 snap -1 2024-04-20T02:51:14.094 INFO:tasks.rados.rados.0.smithi105.stdout:975: expect (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:51:14.094 INFO:tasks.rados.rados.0.smithi105.stdout:976: read oid 20 snap -1 2024-04-20T02:51:14.094 INFO:tasks.rados.rados.0.smithi105.stdout:976: expect (ObjNum 94 snap 3 seq_num 94) 2024-04-20T02:51:14.095 INFO:tasks.rados.rados.0.smithi105.stdout:977: write oid 10 current snap is 10 2024-04-20T02:51:14.095 INFO:tasks.rados.rados.0.smithi105.stdout:977: seq_num 156 ranges {621892=465904,1712485=671711} 2024-04-20T02:51:14.101 INFO:tasks.rados.rados.0.smithi105.stdout:977: writing smithi10548823-10 from 621892 to 1087796 tid 1 2024-04-20T02:51:14.108 INFO:tasks.rados.rados.0.smithi105.stdout:977: writing smithi10548823-10 from 1712485 to 2384196 tid 2 2024-04-20T02:51:14.108 INFO:tasks.rados.rados.0.smithi105.stdout:978: read oid 39 snap -1 2024-04-20T02:51:14.109 INFO:tasks.rados.rados.0.smithi105.stdout:978: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:14.109 INFO:tasks.rados.rados.0.smithi105.stdout:979: snap_create 2024-04-20T02:51:14.109 INFO:tasks.rados.rados.0.smithi105.stdout:967: got (16) Device or resource busy 2024-04-20T02:51:14.109 INFO:tasks.rados.rados.0.smithi105.stdout:check_oldest_snap_flushed oid 41 v 112 (ObjNum 87 snap 2 seq_num 87) unflushed 9 2024-04-20T02:51:14.109 INFO:tasks.rados.rados.0.smithi105.stdout:966: expect (ObjNum 144 snap 9 seq_num 144) 2024-04-20T02:51:14.158 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:14.159+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:14.218 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:14.218 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:14.218 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:14.218 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:14.218 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:14.228 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:14.242 INFO:tasks.rados.rados.0.smithi105.stdout:969: finishing set_chunk to oid 12 2024-04-20T02:51:14.242 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 12 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-04-20T02:51:14.242 INFO:tasks.rados.rados.0.smithi105.stdout:975: expect (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:51:14.259 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:14.259+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:51:14.271 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:51:14.283 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-04-20T02:51:13.456632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":278233030,"num_objects":211,"num_object_clones":35,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":4,"num_read":1669,"num_read_kb":1266632,"num_write":1308,"num_write_kb":350011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":45431190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":54,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":87,"acting":87,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":697604,"kb_used_data":469776,"kb_used_omap":12,"kb_used_meta":224947,"kb_avail":754277116,"statfs":{"total":773094113280,"available":772379766784,"internally_reserved":0,"allocated":481050624,"data_stored":555194509,"data_compressed":12803180,"data_compressed_allocated":13479936,"data_compressed_original":31988855,"omap_allocated":12807,"internal_metadata":230346233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7907732,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":191,"num_read_kb":261919,"num_write":50,"num_write_kb":18764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008356"},"pg_stats":[{"pgid":"4.f","version":"34'15","reported_seq":98,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.449050+0000","last_change":"2024-04-20T02:51:12.449411+0000","last_active":"2024-04-20T02:51:12.449050+0000","last_peered":"2024-04-20T02:51:12.449050+0000","last_clean":"2024-04-20T02:51:12.449050+0000","last_became_active":"2024-04-20T02:50:35.245323+0000","last_became_peered":"2024-04-20T02:50:35.245323+0000","last_unstale":"2024-04-20T02:51:12.449050+0000","last_undegraded":"2024-04-20T02:51:12.449050+0000","last_fullsized":"2024-04-20T02:51:12.449050+0000","mapping_epoch":22,"log_start":"33'10","ondisk_log_start":"33'10","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":10,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:53.742736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99966113099999998,"stat_sum":{"num_bytes":3428595,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":8320,"num_write":14,"num_write_kb":4030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.e","version":"51'30","reported_seq":108,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.898590+0000","last_change":"2024-04-20T02:51:06.133569+0000","last_active":"2024-04-20T02:51:08.898590+0000","last_peered":"2024-04-20T02:51:08.898590+0000","last_clean":"2024-04-20T02:51:08.898590+0000","last_became_active":"2024-04-20T02:50:35.244872+0000","last_became_peered":"2024-04-20T02:50:35.244872+0000","last_unstale":"2024-04-20T02:51:08.898590+0000","last_undegraded":"2024-04-20T02:51:08.898590+0000","last_fullsized":"2024-04-20T02:51:08.898590+0000","mapping_epoch":22,"log_start":"49'27","ondisk_log_start":"49'27","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":27,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:39.148211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014166396,"stat_sum":{"num_bytes":5761953,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":26678,"num_write":20,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.d","version":"51'64","reported_seq":180,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.914763+0000","last_change":"2024-04-20T02:51:06.140431+0000","last_active":"2024-04-20T02:51:08.914763+0000","last_peered":"2024-04-20T02:51:08.914763+0000","last_clean":"2024-04-20T02:51:08.914763+0000","last_became_active":"2024-04-20T02:50:35.528107+0000","last_became_peered":"2024-04-20T02:50:35.528107+0000","last_unstale":"2024-04-20T02:51:08.914763+0000","last_undegraded":"2024-04-20T02:51:08.914763+0000","last_fullsized":"2024-04-20T02:51:08.914763+0000","mapping_epoch":22,"log_start":"49'61","ondisk_log_start":"49'61","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:59.747989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020978734999999998,"stat_sum":{"num_bytes":16925968,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":138,"num_read_kb":157400,"num_write":47,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.c","version":"49'39","reported_seq":118,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.840601+0000","last_change":"2024-04-20T02:51:06.128048+0000","last_active":"2024-04-20T02:51:08.840601+0000","last_peered":"2024-04-20T02:51:08.840601+0000","last_clean":"2024-04-20T02:51:08.840601+0000","last_became_active":"2024-04-20T02:50:35.241480+0000","last_became_peered":"2024-04-20T02:50:35.241480+0000","last_unstale":"2024-04-20T02:51:08.840601+0000","last_undegraded":"2024-04-20T02:51:08.840601+0000","last_fullsized":"2024-04-20T02:51:08.840601+0000","mapping_epoch":22,"log_start":"42'33","ondisk_log_start":"42'33","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":33,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.915698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087151550000000005,"stat_sum":{"num_bytes":7683485,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":31347,"num_write":31,"num_write_kb":8780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.b","version":"52'71","reported_seq":175,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.518160+0000","last_change":"2024-04-20T02:51:06.126734+0000","last_active":"2024-04-20T02:51:09.518160+0000","last_peered":"2024-04-20T02:51:09.518160+0000","last_clean":"2024-04-20T02:51:09.518160+0000","last_became_active":"2024-04-20T02:50:35.242331+0000","last_became_peered":"2024-04-20T02:50:35.242331+0000","last_unstale":"2024-04-20T02:51:09.518160+0000","last_undegraded":"2024-04-20T02:51:09.518160+0000","last_fullsized":"2024-04-20T02:51:09.518160+0000","mapping_epoch":22,"log_start":"48'68","ondisk_log_start":"48'68","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.773051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064970080000000003,"stat_sum":{"num_bytes":14033713,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":94,"num_read_kb":102342,"num_write":56,"num_write_kb":15626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.a","version":"53'33","reported_seq":147,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.469786+0000","last_change":"2024-04-20T02:51:12.470019+0000","last_active":"2024-04-20T02:51:12.469786+0000","last_peered":"2024-04-20T02:51:12.469786+0000","last_clean":"2024-04-20T02:51:12.469786+0000","last_became_active":"2024-04-20T02:50:35.413503+0000","last_became_peered":"2024-04-20T02:50:35.413503+0000","last_unstale":"2024-04-20T02:51:12.469786+0000","last_undegraded":"2024-04-20T02:51:12.469786+0000","last_fullsized":"2024-04-20T02:51:12.469786+0000","mapping_epoch":22,"log_start":"44'29","ondisk_log_start":"44'29","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":29,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:57.185462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014460114,"stat_sum":{"num_bytes":8235364,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":99,"num_read_kb":115384,"num_write":28,"num_write_kb":7354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.9","version":"50'114","reported_seq":248,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.840448+0000","last_change":"2024-04-20T02:51:06.127431+0000","last_active":"2024-04-20T02:51:08.840448+0000","last_peered":"2024-04-20T02:51:08.840448+0000","last_clean":"2024-04-20T02:51:08.840448+0000","last_became_active":"2024-04-20T02:50:35.244955+0000","last_became_peered":"2024-04-20T02:50:35.244955+0000","last_unstale":"2024-04-20T02:51:08.840448+0000","last_undegraded":"2024-04-20T02:51:08.840448+0000","last_fullsized":"2024-04-20T02:51:08.840448+0000","mapping_epoch":22,"log_start":"50'110","ondisk_log_start":"50'110","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":110,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:51.105656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093707679999999998,"stat_sum":{"num_bytes":26183771,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":179,"num_read_kb":183049,"num_write":88,"num_write_kb":25797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.8","version":"48'47","reported_seq":136,"reported_epoch":51,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:08.840468+0000","last_change":"2024-04-20T02:51:02.924920+0000","last_active":"2024-04-20T02:51:08.840468+0000","last_peered":"2024-04-20T02:51:08.840468+0000","last_clean":"2024-04-20T02:50:58.916741+0000","last_became_active":"2024-04-20T02:51:02.922909+0000","last_became_peered":"2024-04-20T02:51:02.922909+0000","last_unstale":"2024-04-20T02:51:08.840468+0000","last_undegraded":"2024-04-20T02:51:08.840468+0000","last_fullsized":"2024-04-20T02:51:08.840468+0000","mapping_epoch":44,"log_start":"42'44","ondisk_log_start":"42'44","created":22,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":44,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.192506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041300199999999999,"stat_sum":{"num_bytes":10144194,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":54887,"num_write":36,"num_write_kb":8549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[4,7,6],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-20T02:51:10.445505+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:51:10.445505+0000","last_peered":"2024-04-20T02:51:10.445505+0000","last_clean":"2024-04-20T02:51:10.445505+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:51:10.445505+0000","last_undegraded":"2024-04-20T02:51:10.445505+0000","last_fullsized":"2024-04-20T02:51:10.445505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":94,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.449497+0000","last_change":"2024-04-20T02:51:12.449685+0000","last_active":"2024-04-20T02:51:12.449497+0000","last_peered":"2024-04-20T02:51:12.449497+0000","last_clean":"2024-04-20T02:51:12.449497+0000","last_became_active":"2024-04-20T02:51:01.129374+0000","last_became_peered":"2024-04-20T02:51:01.129374+0000","last_unstale":"2024-04-20T02:51:12.449497+0000","last_undegraded":"2024-04-20T02:51:12.449497+0000","last_fullsized":"2024-04-20T02:51:12.449497+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:48.451347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99990455700000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.4","version":"54'16","reported_seq":105,"reported_epoch":54,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:12.449764+0000","last_change":"2024-04-20T02:51:12.449764+0000","last_active":"2024-04-20T02:51:12.449764+0000","last_peered":"2024-04-20T02:51:12.449764+0000","last_clean":"2024-04-20T02:51:12.449764+0000","last_became_active":"2024-04-20T02:50:35.528215+0000","last_became_peered":"2024-04-20T02:50:35.528215+0000","last_unstale":"2024-04-20T02:51:12.449764+0000","last_undegraded":"2024-04-20T02:51:12.449764+0000","last_fullsized":"2024-04-20T02:51:12.449764+0000","mapping_epoch":22,"log_start":"49'12","ondisk_log_start":"49'12","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":12,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99860969499999996,"stat_sum":{"num_bytes":3092455,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":53444,"num_write":10,"num_write_kb":2385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.459377+0000","last_change":"2024-04-20T02:51:01.134074+0000","last_active":"2024-04-20T02:51:11.459377+0000","last_peered":"2024-04-20T02:51:11.459377+0000","last_clean":"2024-04-20T02:51:11.459377+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:51:11.459377+0000","last_undegraded":"2024-04-20T02:51:11.459377+0000","last_fullsized":"2024-04-20T02:51:11.459377+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:38.091564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"52'89","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.543472+0000","last_change":"2024-04-20T02:50:34.242061+0000","last_active":"2024-04-20T02:51:09.543472+0000","last_peered":"2024-04-20T02:51:09.543472+0000","last_clean":"2024-04-20T02:51:09.543472+0000","last_became_active":"2024-04-20T02:50:34.241638+0000","last_became_peered":"2024-04-20T02:50:34.241638+0000","last_unstale":"2024-04-20T02:51:09.543472+0000","last_undegraded":"2024-04-20T02:51:09.543472+0000","last_fullsized":"2024-04-20T02:51:09.543472+0000","mapping_epoch":21,"log_start":"46'86","ondisk_log_start":"46'86","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":92,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:58.050023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27403590,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":82,"num_read_kb":3703,"num_write":114,"num_write_kb":34796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"53'38","reported_seq":174,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.451318+0000","last_change":"2024-04-20T02:51:12.451512+0000","last_active":"2024-04-20T02:51:12.451318+0000","last_peered":"2024-04-20T02:51:12.451318+0000","last_clean":"2024-04-20T02:51:12.451318+0000","last_became_active":"2024-04-20T02:51:04.417108+0000","last_became_peered":"2024-04-20T02:51:04.417108+0000","last_unstale":"2024-04-20T02:51:12.451318+0000","last_undegraded":"2024-04-20T02:51:12.451318+0000","last_fullsized":"2024-04-20T02:51:12.451318+0000","mapping_epoch":46,"log_start":"42'33","ondisk_log_start":"42'33","created":22,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":33,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:00.545359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.000954766,"stat_sum":{"num_bytes":7640560,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":54652,"num_write":33,"num_write_kb":9882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5515811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.464845+0000","last_change":"2024-04-20T02:50:28.091742+0000","last_active":"2024-04-20T02:51:11.464845+0000","last_peered":"2024-04-20T02:51:11.464845+0000","last_clean":"2024-04-20T02:51:11.464845+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:51:11.464845+0000","last_undegraded":"2024-04-20T02:51:11.464845+0000","last_fullsized":"2024-04-20T02:51:11.464845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:32.326186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"51'93","reported_seq":142,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-20T02:51:10.445074+0000","last_change":"2024-04-20T02:50:34.241909+0000","last_active":"2024-04-20T02:51:10.445074+0000","last_peered":"2024-04-20T02:51:10.445074+0000","last_clean":"2024-04-20T02:51:10.445074+0000","last_became_active":"2024-04-20T02:50:34.241613+0000","last_became_peered":"2024-04-20T02:50:34.241613+0000","last_unstale":"2024-04-20T02:51:10.445074+0000","last_undegraded":"2024-04-20T02:51:10.445074+0000","last_fullsized":"2024-04-20T02:51:10.445074+0000","mapping_epoch":21,"log_start":"45'90","ondisk_log_start":"45'90","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":102,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:54.710965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19924935,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":82,"num_read_kb":1868,"num_write":122,"num_write_kb":36775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'32","reported_seq":135,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.469721+0000","last_change":"2024-04-20T02:51:12.469951+0000","last_active":"2024-04-20T02:51:12.469721+0000","last_peered":"2024-04-20T02:51:12.469721+0000","last_clean":"2024-04-20T02:51:12.469721+0000","last_became_active":"2024-04-20T02:50:35.413590+0000","last_became_peered":"2024-04-20T02:50:35.413590+0000","last_unstale":"2024-04-20T02:51:12.469721+0000","last_undegraded":"2024-04-20T02:51:12.469721+0000","last_fullsized":"2024-04-20T02:51:12.469721+0000","mapping_epoch":22,"log_start":"42'28","ondisk_log_start":"42'28","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":28,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:44.785443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014352482999999999,"stat_sum":{"num_bytes":9479991,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":88948,"num_write":23,"num_write_kb":7717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.459300+0000","last_change":"2024-04-20T02:51:01.151894+0000","last_active":"2024-04-20T02:51:11.459300+0000","last_peered":"2024-04-20T02:51:11.459300+0000","last_clean":"2024-04-20T02:51:11.459300+0000","last_became_active":"2024-04-20T02:51:01.151544+0000","last_became_peered":"2024-04-20T02:51:01.151544+0000","last_unstale":"2024-04-20T02:51:11.459300+0000","last_undegraded":"2024-04-20T02:51:11.459300+0000","last_fullsized":"2024-04-20T02:51:11.459300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:29.245779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.899914+0000","last_change":"2024-04-20T02:50:19.923431+0000","last_active":"2024-04-20T02:51:08.899914+0000","last_peered":"2024-04-20T02:51:08.899914+0000","last_clean":"2024-04-20T02:51:08.899914+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:51:08.899914+0000","last_undegraded":"2024-04-20T02:51:08.899914+0000","last_fullsized":"2024-04-20T02:51:08.899914+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:50:17.569030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.655607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"53'105","reported_seq":245,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.447735+0000","last_change":"2024-04-20T02:51:07.118974+0000","last_active":"2024-04-20T02:51:11.447735+0000","last_peered":"2024-04-20T02:51:11.447735+0000","last_clean":"2024-04-20T02:51:11.447735+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:51:11.447735+0000","last_undegraded":"2024-04-20T02:51:11.447735+0000","last_fullsized":"2024-04-20T02:51:11.447735+0000","mapping_epoch":49,"log_start":"52'102","ondisk_log_start":"52'102","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":110,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:42.672805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27256345,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":92,"num_read_kb":5490,"num_write":122,"num_write_kb":50638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"52'43","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.470545+0000","last_change":"2024-04-20T02:51:06.126486+0000","last_active":"2024-04-20T02:51:09.470545+0000","last_peered":"2024-04-20T02:51:09.470545+0000","last_clean":"2024-04-20T02:51:09.470545+0000","last_became_active":"2024-04-20T02:50:35.528177+0000","last_became_peered":"2024-04-20T02:50:35.528177+0000","last_unstale":"2024-04-20T02:51:09.470545+0000","last_undegraded":"2024-04-20T02:51:09.470545+0000","last_fullsized":"2024-04-20T02:51:09.470545+0000","mapping_epoch":22,"log_start":"49'39","ondisk_log_start":"49'39","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":39,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:55.042511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065151180000000003,"stat_sum":{"num_bytes":12790883,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":78,"num_read_kb":73062,"num_write":53,"num_write_kb":13339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.899929+0000","last_change":"2024-04-20T02:50:28.092115+0000","last_active":"2024-04-20T02:51:08.899929+0000","last_peered":"2024-04-20T02:51:08.899929+0000","last_clean":"2024-04-20T02:51:08.899929+0000","last_became_active":"2024-04-20T02:50:28.091570+0000","last_became_peered":"2024-04-20T02:50:28.091570+0000","last_unstale":"2024-04-20T02:51:08.899929+0000","last_undegraded":"2024-04-20T02:51:08.899929+0000","last_fullsized":"2024-04-20T02:51:08.899929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:27.403210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"48'116","reported_seq":174,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.792164+0000","last_change":"2024-04-20T02:50:34.239216+0000","last_active":"2024-04-20T02:51:08.792164+0000","last_peered":"2024-04-20T02:51:08.792164+0000","last_clean":"2024-04-20T02:51:08.792164+0000","last_became_active":"2024-04-20T02:50:34.238906+0000","last_became_peered":"2024-04-20T02:50:34.238906+0000","last_unstale":"2024-04-20T02:51:08.792164+0000","last_undegraded":"2024-04-20T02:51:08.792164+0000","last_fullsized":"2024-04-20T02:51:08.792164+0000","mapping_epoch":21,"log_start":"42'113","ondisk_log_start":"42'113","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":138,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.390119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28959845,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":111,"num_read_kb":6350,"num_write":143,"num_write_kb":48210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"52'127","reported_seq":297,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.436361+0000","last_change":"2024-04-20T02:51:06.140804+0000","last_active":"2024-04-20T02:51:09.436361+0000","last_peered":"2024-04-20T02:51:09.436361+0000","last_clean":"2024-04-20T02:51:09.436361+0000","last_became_active":"2024-04-20T02:51:05.421483+0000","last_became_peered":"2024-04-20T02:51:05.421483+0000","last_unstale":"2024-04-20T02:51:09.436361+0000","last_undegraded":"2024-04-20T02:51:09.436361+0000","last_fullsized":"2024-04-20T02:51:09.436361+0000","mapping_epoch":47,"log_start":"49'124","ondisk_log_start":"49'124","created":22,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.981337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020740003999999999,"stat_sum":{"num_bytes":23383373,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":139,"num_read_kb":154428,"num_write":95,"num_write_kb":24765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":18280879,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.898181+0000","last_change":"2024-04-20T02:50:28.091938+0000","last_active":"2024-04-20T02:51:08.898181+0000","last_peered":"2024-04-20T02:51:08.898181+0000","last_clean":"2024-04-20T02:51:08.898181+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:51:08.898181+0000","last_undegraded":"2024-04-20T02:51:08.898181+0000","last_fullsized":"2024-04-20T02:51:08.898181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:34.990718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-20T02:51:10.445522+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:51:10.445522+0000","last_peered":"2024-04-20T02:51:10.445522+0000","last_clean":"2024-04-20T02:51:10.445522+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:51:10.445522+0000","last_undegraded":"2024-04-20T02:51:10.445522+0000","last_fullsized":"2024-04-20T02:51:10.445522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"54'67","reported_seq":185,"reported_epoch":54,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:12.452284+0000","last_change":"2024-04-20T02:51:12.450055+0000","last_active":"2024-04-20T02:51:12.452284+0000","last_peered":"2024-04-20T02:51:12.452284+0000","last_clean":"2024-04-20T02:51:12.452284+0000","last_became_active":"2024-04-20T02:50:35.413445+0000","last_became_peered":"2024-04-20T02:50:35.413445+0000","last_unstale":"2024-04-20T02:51:12.452284+0000","last_undegraded":"2024-04-20T02:51:12.452284+0000","last_fullsized":"2024-04-20T02:51:12.452284+0000","mapping_epoch":22,"log_start":"51'61","ondisk_log_start":"51'61","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":61,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:56.404298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0023455880000001,"stat_sum":{"num_bytes":10840416,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":105,"num_read_kb":85234,"num_write":44,"num_write_kb":11536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.450251+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:51:11.450251+0000","last_peered":"2024-04-20T02:51:11.450251+0000","last_clean":"2024-04-20T02:51:11.450251+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:51:11.450251+0000","last_undegraded":"2024-04-20T02:51:11.450251+0000","last_fullsized":"2024-04-20T02:51:11.450251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.432280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"52'79","reported_seq":218,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.549410+0000","last_change":"2024-04-20T02:51:06.119828+0000","last_active":"2024-04-20T02:51:09.549410+0000","last_peered":"2024-04-20T02:51:09.549410+0000","last_clean":"2024-04-20T02:51:09.549410+0000","last_became_active":"2024-04-20T02:51:04.413969+0000","last_became_peered":"2024-04-20T02:51:04.413969+0000","last_unstale":"2024-04-20T02:51:09.549410+0000","last_undegraded":"2024-04-20T02:51:09.549410+0000","last_fullsized":"2024-04-20T02:51:09.549410+0000","mapping_epoch":46,"log_start":"52'76","ondisk_log_start":"52'76","created":22,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":76,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.956090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071174699999999999,"stat_sum":{"num_bytes":14473226,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":70,"num_read_kb":59946,"num_write":70,"num_write_kb":18690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7407305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":174097947,"num_objects":82,"num_object_clones":35,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":4,"num_read":1186,"num_read_kb":1249121,"num_write":648,"num_write_kb":177176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":31203995,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":54,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":318308352,"data_stored":381035946,"data_compressed":1651560,"data_compressed_allocated":1400832,"data_compressed_original":4062327,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":48,"acting":48,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":103544715,"num_objects":127,"num_object_clones":0,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":367,"num_read_kb":17411,"num_write":501,"num_write_kb":170419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":161460224,"data_stored":175351286,"data_compressed":11184321,"data_compressed_allocated":11980800,"data_compressed_original":26456064,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509456,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67260,"kb_used_data":40168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94304580,"statfs":{"total":96636764160,"available":96567889920,"internally_reserved":0,"allocated":41132032,"data_stored":52309410,"data_compressed":16844,"data_compressed_allocated":45056,"data_compressed_original":577536,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757253,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75860,"kb_used_data":45520,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94295980,"statfs":{"total":96636764160,"available":96559083520,"internally_reserved":0,"allocated":46612480,"data_stored":52416628,"data_compressed":1083307,"data_compressed_allocated":1159168,"data_compressed_original":2473984,"omap_allocated":1657,"internal_metadata":28375431},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56988,"kb_used_data":29968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94314852,"statfs":{"total":96636764160,"available":96578408448,"internally_reserved":0,"allocated":30687232,"data_stored":36085808,"data_compressed":1041091,"data_compressed_allocated":1048576,"data_compressed_original":2203648,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91444,"kb_used_data":64424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94280396,"statfs":{"total":96636764160,"available":96543125504,"internally_reserved":0,"allocated":65970176,"data_stored":75138624,"data_compressed":1361937,"data_compressed_allocated":1327104,"data_compressed_original":3099005,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":54088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":55386112,"data_stored":63810496,"data_compressed":2040256,"data_compressed_allocated":2162688,"data_compressed_original":4795990,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.217}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98756,"kb_used_data":71728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94273084,"statfs":{"total":96636764160,"available":96535638016,"internally_reserved":0,"allocated":73449472,"data_stored":80310869,"data_compressed":3034714,"data_compressed_allocated":3219456,"data_compressed_original":7592743,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509452,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101592,"kb_used_data":70400,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94270248,"statfs":{"total":96636764160,"available":96532733952,"internally_reserved":0,"allocated":72089600,"data_stored":84378489,"data_compressed":2095144,"data_compressed_allocated":2289664,"data_compressed_original":5824512,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509454,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":124596,"kb_used_data":93480,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94247244,"statfs":{"total":96636764160,"available":96509177856,"internally_reserved":0,"allocated":95723520,"data_stored":110744185,"data_compressed":2129887,"data_compressed_allocated":2228224,"data_compressed_original":5421437,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28381184,"data_stored":30949639,"data_compressed":1843809,"data_compressed_allocated":1990656,"data_compressed_original":4657152,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11038720,"data_stored":11916936,"data_compressed":822943,"data_compressed_allocated":872448,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11038720,"data_stored":11916936,"data_compressed":822943,"data_compressed_allocated":872448,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28123136,"data_stored":30409704,"data_compressed":1915052,"data_compressed_allocated":2048000,"data_compressed_original":4419584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28381184,"data_stored":30949639,"data_compressed":1843809,"data_compressed_allocated":1990656,"data_compressed_original":4657152,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14135296,"data_stored":15322309,"data_compressed":1048539,"data_compressed_allocated":1118208,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40361984,"data_stored":43886123,"data_compressed":2887226,"data_compressed_allocated":3088384,"data_compressed_original":6758400,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44679168,"data_stored":55212658,"data_compressed":294672,"data_compressed_allocated":266240,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54341632,"data_stored":62977902,"data_compressed":538994,"data_compressed_allocated":454656,"data_compressed_original":1288573,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19058688,"data_stored":23925086,"data_compressed":218148,"data_compressed_allocated":176128,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26673152,"data_stored":33157006,"data_compressed":125204,"data_compressed_allocated":114688,"data_compressed_original":376406,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":68710400,"data_stored":81713216,"data_compressed":286078,"data_compressed_allocated":237568,"data_compressed_original":764285,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":31875072,"data_stored":36844968,"data_compressed":34768,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32444416,"data_stored":35652032,"data_compressed":142170,"data_compressed_allocated":98304,"data_compressed_original":322343,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40525824,"data_stored":51553078,"data_compressed":11526,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:51:14.285 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-20T02:51:14.285 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:51:14.285 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-20T02:51:14.285 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:51:14.360 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:14.361+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:14.400 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:14.400 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:14.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:14.400 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:14.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.400 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 54.795676481000001, 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:30 4.8 4:1c68e542:::smithi10548823-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.035524293000002, 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033039351000000002, 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.407 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 30 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364066+0000", 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364312+0000", 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000246708 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364328+0000", 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5376999999999999e-05 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364339+0000", 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.092e-05 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375692+0000", 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011353103999999999 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375805+0000", 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000113414 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375970+0000", 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016508399999999999 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388615+0000", 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012645145 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391156+0000", 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025406410000000002 2024-04-20T02:51:14.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396990+0000", 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0058340390000000001 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397042+0000", 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2454000000000003e-05 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397106+0000", 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3632000000000003e-05 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.412 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:31 4.8 4:1c68e542:::smithi10548823-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.035237596999998, 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032910542000000001, 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 31 2024-04-20T02:51:14.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:14.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364357+0000", 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1160000000000001e-06 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364363+0000", 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1680000000000001e-06 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364367+0000", 2024-04-20T02:51:14.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8750000000000002e-06 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388814+0000", 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024447437999999998 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388914+0000", 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9557999999999998e-05 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389062+0000", 2024-04-20T02:51:14.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014799600000000001 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391160+0000", 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020979760000000001 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396038+0000", 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048784689999999999 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397155+0000", 2024-04-20T02:51:14.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001116286 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397208+0000", 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3273000000000003e-05 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397264+0000", 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5962999999999999e-05 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:32 4.8 4:1c68e542:::smithi10548823-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.035218882999999, 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033121840999999999, 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 32 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:14.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364375+0000", 2024-04-20T02:51:14.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1149999999999998e-06 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364382+0000", 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4259999999999997e-06 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364387+0000", 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6129999999999997e-06 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389813+0000", 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025426621 2024-04-20T02:51:14.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389905+0000", 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2011999999999999e-05 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397299+0000", 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007393897 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397395+0000", 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5293999999999999e-05 2024-04-20T02:51:14.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397494+0000", 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9289000000000004e-05 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.031810082, 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.423 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:14.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:51:14.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:51:14.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.918507040999998, 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:51:14.427 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:14.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:51:14.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:51:14.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:66 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.913149152999999, 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033365526, 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.433 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 66 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486438+0000", 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486778+0000", 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000339709 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486798+0000", 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0462000000000002e-05 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486812+0000", 2024-04-20T02:51:14.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3589e-05 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496889+0000", 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010077477 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496978+0000", 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8205e-05 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.497125+0000", 2024-04-20T02:51:14.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000147665 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518444+0000", 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.021318713 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519407+0000", 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00096312999999999998 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519727+0000", 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031953500000000002 2024-04-20T02:51:14.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519764+0000", 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7406000000000001e-05 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519807+0000", 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3402000000000002e-05 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:67 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2268912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.912763968999997, 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033307638000000001, 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 67 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486826+0000", 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486831+0000", 2024-04-20T02:51:14.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0189999999999997e-06 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486839+0000", 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4710000000000001e-06 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486844+0000", 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.57e-06 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509240+0000", 2024-04-20T02:51:14.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022395241 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509338+0000", 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8034000000000004e-05 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509498+0000", 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015993999999999999 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518471+0000", 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0089732730000000004 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519859+0000", 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013877130000000001 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520042+0000", 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182981 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520084+0000", 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2518999999999997e-05 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520135+0000", 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0491000000000002e-05 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.912741523999998, 2024-04-20T02:51:14.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:161 4.3 e29/27 4:c32272c0:::smithi10548823-41:head v 29'16, mlcod=29'14)", 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.565305432999999, 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031990785000000001, 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834285+0000", 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835168+0000", 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00088287800000000003 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835180+0000", 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2340999999999999e-05 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835187+0000", 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7830000000000001e-06 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835227+0000", 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0754000000000003e-05 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835243+0000", 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5607000000000002e-05 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866234+0000", 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030990891 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866276+0000", 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2469999999999998e-05 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.558721388999999, 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.548357783999997, 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.457 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:14.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:51:14.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.540273304999999, 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.461 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:51:14.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:51:14.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:981 4.7 e32/27 4:e9876a63:::smithi10548823-30:head v 32'19, mlcod=30'17)", 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:14.465 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.810220830999999, 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033092985999999998, 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589369+0000", 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590134+0000", 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076479 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590148+0000", 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3679999999999999e-05 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590157+0000", 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590209+0000", 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2237999999999997e-05 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590235+0000", 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6251e-05 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622430+0000", 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032194953999999998 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622463+0000", 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2780000000000001e-05 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1230 4.5 e38/22 4:a94dc531:::smithi10548823-32:head v 38'11, mlcod=29'8)", 2024-04-20T02:51:14.469 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.987075433000001, 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033312785999999997, 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412515+0000", 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412827+0000", 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031200999999999999 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412835+0000", 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.61e-06 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412842+0000", 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6990000000000003e-06 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412921+0000", 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8380000000000005e-05 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412944+0000", 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3609e-05 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445795+0000", 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032850279000000003 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445828+0000", 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3673000000000001e-05 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.25871508, 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.25357797, 2024-04-20T02:51:14.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:51:14.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.253304957999999, 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:51:14.485 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:51:14.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:51:14.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.253275573, 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:14.491 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 3.944381608, 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035060916999999997, 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:14.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456603+0000", 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001394295 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:14.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456616+0000", 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2605999999999999e-05 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456631+0000", 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4735e-05 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456682+0000", 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1048000000000003e-05 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:14.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456705+0000", 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2881e-05 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490235+0000", 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033530250999999997 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490270+0000", 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5379000000000001e-05 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:14.499 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:14.500 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:14.501 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:14.501+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:14.501 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:14.525 INFO:tasks.rados.rados.0.smithi105.stdout:978: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:14.594 INFO:tasks.rados.rados.0.smithi105.stdout:976: expect (ObjNum 94 snap 3 seq_num 94) 2024-04-20T02:51:14.602 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:14.603+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:14.662 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:51:14.664 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:51:14.671 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:14.671 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:14.671 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:14.671 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:14.675 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-20T02:51:14.466795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":278233030,"num_objects":211,"num_object_clones":35,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":4,"num_read":1669,"num_read_kb":1266632,"num_write":1308,"num_write_kb":350011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":45431190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":54,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":87,"acting":87,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":697604,"kb_used_data":469776,"kb_used_omap":12,"kb_used_meta":224947,"kb_avail":754277116,"statfs":{"total":773094113280,"available":772379766784,"internally_reserved":0,"allocated":481050624,"data_stored":555194509,"data_compressed":12803180,"data_compressed_allocated":13479936,"data_compressed_original":31988855,"omap_allocated":12807,"internal_metadata":230346233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3838532,"num_objects":-3,"num_object_clones":-3,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":-1,"num_read":75,"num_read_kb":90932,"num_write":8,"num_write_kb":7480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.024282"},"pg_stats":[{"pgid":"4.f","version":"34'15","reported_seq":98,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.449050+0000","last_change":"2024-04-20T02:51:12.449411+0000","last_active":"2024-04-20T02:51:12.449050+0000","last_peered":"2024-04-20T02:51:12.449050+0000","last_clean":"2024-04-20T02:51:12.449050+0000","last_became_active":"2024-04-20T02:50:35.245323+0000","last_became_peered":"2024-04-20T02:50:35.245323+0000","last_unstale":"2024-04-20T02:51:12.449050+0000","last_undegraded":"2024-04-20T02:51:12.449050+0000","last_fullsized":"2024-04-20T02:51:12.449050+0000","mapping_epoch":22,"log_start":"33'10","ondisk_log_start":"33'10","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":10,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:53.742736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99966113099999998,"stat_sum":{"num_bytes":3428595,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":8320,"num_write":14,"num_write_kb":4030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.e","version":"51'30","reported_seq":108,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.898590+0000","last_change":"2024-04-20T02:51:06.133569+0000","last_active":"2024-04-20T02:51:08.898590+0000","last_peered":"2024-04-20T02:51:08.898590+0000","last_clean":"2024-04-20T02:51:08.898590+0000","last_became_active":"2024-04-20T02:50:35.244872+0000","last_became_peered":"2024-04-20T02:50:35.244872+0000","last_unstale":"2024-04-20T02:51:08.898590+0000","last_undegraded":"2024-04-20T02:51:08.898590+0000","last_fullsized":"2024-04-20T02:51:08.898590+0000","mapping_epoch":22,"log_start":"49'27","ondisk_log_start":"49'27","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":27,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:39.148211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014166396,"stat_sum":{"num_bytes":5761953,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":26678,"num_write":20,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.d","version":"51'64","reported_seq":180,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.914763+0000","last_change":"2024-04-20T02:51:06.140431+0000","last_active":"2024-04-20T02:51:08.914763+0000","last_peered":"2024-04-20T02:51:08.914763+0000","last_clean":"2024-04-20T02:51:08.914763+0000","last_became_active":"2024-04-20T02:50:35.528107+0000","last_became_peered":"2024-04-20T02:50:35.528107+0000","last_unstale":"2024-04-20T02:51:08.914763+0000","last_undegraded":"2024-04-20T02:51:08.914763+0000","last_fullsized":"2024-04-20T02:51:08.914763+0000","mapping_epoch":22,"log_start":"49'61","ondisk_log_start":"49'61","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:59.747989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020978734999999998,"stat_sum":{"num_bytes":16925968,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":138,"num_read_kb":157400,"num_write":47,"num_write_kb":12536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.c","version":"49'39","reported_seq":118,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.840601+0000","last_change":"2024-04-20T02:51:06.128048+0000","last_active":"2024-04-20T02:51:08.840601+0000","last_peered":"2024-04-20T02:51:08.840601+0000","last_clean":"2024-04-20T02:51:08.840601+0000","last_became_active":"2024-04-20T02:50:35.241480+0000","last_became_peered":"2024-04-20T02:50:35.241480+0000","last_unstale":"2024-04-20T02:51:08.840601+0000","last_undegraded":"2024-04-20T02:51:08.840601+0000","last_fullsized":"2024-04-20T02:51:08.840601+0000","mapping_epoch":22,"log_start":"42'33","ondisk_log_start":"42'33","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":33,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.915698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087151550000000005,"stat_sum":{"num_bytes":7683485,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":31347,"num_write":31,"num_write_kb":8780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.b","version":"52'71","reported_seq":175,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.518160+0000","last_change":"2024-04-20T02:51:06.126734+0000","last_active":"2024-04-20T02:51:09.518160+0000","last_peered":"2024-04-20T02:51:09.518160+0000","last_clean":"2024-04-20T02:51:09.518160+0000","last_became_active":"2024-04-20T02:50:35.242331+0000","last_became_peered":"2024-04-20T02:50:35.242331+0000","last_unstale":"2024-04-20T02:51:09.518160+0000","last_undegraded":"2024-04-20T02:51:09.518160+0000","last_fullsized":"2024-04-20T02:51:09.518160+0000","mapping_epoch":22,"log_start":"48'68","ondisk_log_start":"48'68","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.773051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064970080000000003,"stat_sum":{"num_bytes":14033713,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":94,"num_read_kb":102342,"num_write":56,"num_write_kb":15626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.a","version":"53'33","reported_seq":147,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.469786+0000","last_change":"2024-04-20T02:51:12.470019+0000","last_active":"2024-04-20T02:51:12.469786+0000","last_peered":"2024-04-20T02:51:12.469786+0000","last_clean":"2024-04-20T02:51:12.469786+0000","last_became_active":"2024-04-20T02:50:35.413503+0000","last_became_peered":"2024-04-20T02:50:35.413503+0000","last_unstale":"2024-04-20T02:51:12.469786+0000","last_undegraded":"2024-04-20T02:51:12.469786+0000","last_fullsized":"2024-04-20T02:51:12.469786+0000","mapping_epoch":22,"log_start":"44'29","ondisk_log_start":"44'29","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":29,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:57.185462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014460114,"stat_sum":{"num_bytes":8235364,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":99,"num_read_kb":115384,"num_write":28,"num_write_kb":7354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.9","version":"50'114","reported_seq":248,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.840448+0000","last_change":"2024-04-20T02:51:06.127431+0000","last_active":"2024-04-20T02:51:08.840448+0000","last_peered":"2024-04-20T02:51:08.840448+0000","last_clean":"2024-04-20T02:51:08.840448+0000","last_became_active":"2024-04-20T02:50:35.244955+0000","last_became_peered":"2024-04-20T02:50:35.244955+0000","last_unstale":"2024-04-20T02:51:08.840448+0000","last_undegraded":"2024-04-20T02:51:08.840448+0000","last_fullsized":"2024-04-20T02:51:08.840448+0000","mapping_epoch":22,"log_start":"50'110","ondisk_log_start":"50'110","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":110,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:51.105656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093707679999999998,"stat_sum":{"num_bytes":26183771,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":179,"num_read_kb":183049,"num_write":88,"num_write_kb":25797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"4.8","version":"48'47","reported_seq":136,"reported_epoch":51,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:08.840468+0000","last_change":"2024-04-20T02:51:02.924920+0000","last_active":"2024-04-20T02:51:08.840468+0000","last_peered":"2024-04-20T02:51:08.840468+0000","last_clean":"2024-04-20T02:50:58.916741+0000","last_became_active":"2024-04-20T02:51:02.922909+0000","last_became_peered":"2024-04-20T02:51:02.922909+0000","last_unstale":"2024-04-20T02:51:08.840468+0000","last_undegraded":"2024-04-20T02:51:08.840468+0000","last_fullsized":"2024-04-20T02:51:08.840468+0000","mapping_epoch":44,"log_start":"42'44","ondisk_log_start":"42'44","created":22,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":44,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.192506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041300199999999999,"stat_sum":{"num_bytes":10144194,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":54887,"num_write":36,"num_write_kb":8549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[4,7,6],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-20T02:51:10.445505+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:51:10.445505+0000","last_peered":"2024-04-20T02:51:10.445505+0000","last_clean":"2024-04-20T02:51:10.445505+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:51:10.445505+0000","last_undegraded":"2024-04-20T02:51:10.445505+0000","last_fullsized":"2024-04-20T02:51:10.445505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":94,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.449497+0000","last_change":"2024-04-20T02:51:12.449685+0000","last_active":"2024-04-20T02:51:12.449497+0000","last_peered":"2024-04-20T02:51:12.449497+0000","last_clean":"2024-04-20T02:51:12.449497+0000","last_became_active":"2024-04-20T02:51:01.129374+0000","last_became_peered":"2024-04-20T02:51:01.129374+0000","last_unstale":"2024-04-20T02:51:12.449497+0000","last_undegraded":"2024-04-20T02:51:12.449497+0000","last_fullsized":"2024-04-20T02:51:12.449497+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:48.451347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99990455700000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.4","version":"54'16","reported_seq":105,"reported_epoch":54,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:12.449764+0000","last_change":"2024-04-20T02:51:12.449764+0000","last_active":"2024-04-20T02:51:12.449764+0000","last_peered":"2024-04-20T02:51:12.449764+0000","last_clean":"2024-04-20T02:51:12.449764+0000","last_became_active":"2024-04-20T02:50:35.528215+0000","last_became_peered":"2024-04-20T02:50:35.528215+0000","last_unstale":"2024-04-20T02:51:12.449764+0000","last_undegraded":"2024-04-20T02:51:12.449764+0000","last_fullsized":"2024-04-20T02:51:12.449764+0000","mapping_epoch":22,"log_start":"49'12","ondisk_log_start":"49'12","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":12,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99860969499999996,"stat_sum":{"num_bytes":3092455,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":53444,"num_write":10,"num_write_kb":2385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.459377+0000","last_change":"2024-04-20T02:51:01.134074+0000","last_active":"2024-04-20T02:51:11.459377+0000","last_peered":"2024-04-20T02:51:11.459377+0000","last_clean":"2024-04-20T02:51:11.459377+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:51:11.459377+0000","last_undegraded":"2024-04-20T02:51:11.459377+0000","last_fullsized":"2024-04-20T02:51:11.459377+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:38.091564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"52'89","reported_seq":139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.543472+0000","last_change":"2024-04-20T02:50:34.242061+0000","last_active":"2024-04-20T02:51:09.543472+0000","last_peered":"2024-04-20T02:51:09.543472+0000","last_clean":"2024-04-20T02:51:09.543472+0000","last_became_active":"2024-04-20T02:50:34.241638+0000","last_became_peered":"2024-04-20T02:50:34.241638+0000","last_unstale":"2024-04-20T02:51:09.543472+0000","last_undegraded":"2024-04-20T02:51:09.543472+0000","last_fullsized":"2024-04-20T02:51:09.543472+0000","mapping_epoch":21,"log_start":"46'86","ondisk_log_start":"46'86","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":92,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:58.050023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27403590,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":82,"num_read_kb":3703,"num_write":114,"num_write_kb":34796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"53'38","reported_seq":174,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.451318+0000","last_change":"2024-04-20T02:51:12.451512+0000","last_active":"2024-04-20T02:51:12.451318+0000","last_peered":"2024-04-20T02:51:12.451318+0000","last_clean":"2024-04-20T02:51:12.451318+0000","last_became_active":"2024-04-20T02:51:04.417108+0000","last_became_peered":"2024-04-20T02:51:04.417108+0000","last_unstale":"2024-04-20T02:51:12.451318+0000","last_undegraded":"2024-04-20T02:51:12.451318+0000","last_fullsized":"2024-04-20T02:51:12.451318+0000","mapping_epoch":46,"log_start":"42'33","ondisk_log_start":"42'33","created":22,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":33,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:00.545359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.000954766,"stat_sum":{"num_bytes":7640560,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":54652,"num_write":33,"num_write_kb":9882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5515811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.464845+0000","last_change":"2024-04-20T02:50:28.091742+0000","last_active":"2024-04-20T02:51:11.464845+0000","last_peered":"2024-04-20T02:51:11.464845+0000","last_clean":"2024-04-20T02:51:11.464845+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:51:11.464845+0000","last_undegraded":"2024-04-20T02:51:11.464845+0000","last_fullsized":"2024-04-20T02:51:11.464845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:32.326186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"51'93","reported_seq":142,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-20T02:51:10.445074+0000","last_change":"2024-04-20T02:50:34.241909+0000","last_active":"2024-04-20T02:51:10.445074+0000","last_peered":"2024-04-20T02:51:10.445074+0000","last_clean":"2024-04-20T02:51:10.445074+0000","last_became_active":"2024-04-20T02:50:34.241613+0000","last_became_peered":"2024-04-20T02:50:34.241613+0000","last_unstale":"2024-04-20T02:51:10.445074+0000","last_undegraded":"2024-04-20T02:51:10.445074+0000","last_fullsized":"2024-04-20T02:51:10.445074+0000","mapping_epoch":21,"log_start":"45'90","ondisk_log_start":"45'90","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":102,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:54.710965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19924935,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":82,"num_read_kb":1868,"num_write":122,"num_write_kb":36775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"53'32","reported_seq":135,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-20T02:51:12.469721+0000","last_change":"2024-04-20T02:51:12.469951+0000","last_active":"2024-04-20T02:51:12.469721+0000","last_peered":"2024-04-20T02:51:12.469721+0000","last_clean":"2024-04-20T02:51:12.469721+0000","last_became_active":"2024-04-20T02:50:35.413590+0000","last_became_peered":"2024-04-20T02:50:35.413590+0000","last_unstale":"2024-04-20T02:51:12.469721+0000","last_undegraded":"2024-04-20T02:51:12.469721+0000","last_fullsized":"2024-04-20T02:51:12.469721+0000","mapping_epoch":22,"log_start":"42'28","ondisk_log_start":"42'28","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":28,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:44.785443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014352482999999999,"stat_sum":{"num_bytes":9479991,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":88948,"num_write":23,"num_write_kb":7717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.459300+0000","last_change":"2024-04-20T02:51:01.151894+0000","last_active":"2024-04-20T02:51:11.459300+0000","last_peered":"2024-04-20T02:51:11.459300+0000","last_clean":"2024-04-20T02:51:11.459300+0000","last_became_active":"2024-04-20T02:51:01.151544+0000","last_became_peered":"2024-04-20T02:51:01.151544+0000","last_unstale":"2024-04-20T02:51:11.459300+0000","last_undegraded":"2024-04-20T02:51:11.459300+0000","last_fullsized":"2024-04-20T02:51:11.459300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:29.245779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.899914+0000","last_change":"2024-04-20T02:50:19.923431+0000","last_active":"2024-04-20T02:51:08.899914+0000","last_peered":"2024-04-20T02:51:08.899914+0000","last_clean":"2024-04-20T02:51:08.899914+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:51:08.899914+0000","last_undegraded":"2024-04-20T02:51:08.899914+0000","last_fullsized":"2024-04-20T02:51:08.899914+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:50:17.569030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.655607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"53'105","reported_seq":245,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.447735+0000","last_change":"2024-04-20T02:51:07.118974+0000","last_active":"2024-04-20T02:51:11.447735+0000","last_peered":"2024-04-20T02:51:11.447735+0000","last_clean":"2024-04-20T02:51:11.447735+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:51:11.447735+0000","last_undegraded":"2024-04-20T02:51:11.447735+0000","last_fullsized":"2024-04-20T02:51:11.447735+0000","mapping_epoch":49,"log_start":"52'102","ondisk_log_start":"52'102","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":110,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:42.672805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27256345,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":92,"num_read_kb":5490,"num_write":122,"num_write_kb":50638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"52'43","reported_seq":144,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.470545+0000","last_change":"2024-04-20T02:51:06.126486+0000","last_active":"2024-04-20T02:51:09.470545+0000","last_peered":"2024-04-20T02:51:09.470545+0000","last_clean":"2024-04-20T02:51:09.470545+0000","last_became_active":"2024-04-20T02:50:35.528177+0000","last_became_peered":"2024-04-20T02:50:35.528177+0000","last_unstale":"2024-04-20T02:51:09.470545+0000","last_undegraded":"2024-04-20T02:51:09.470545+0000","last_fullsized":"2024-04-20T02:51:09.470545+0000","mapping_epoch":22,"log_start":"49'39","ondisk_log_start":"49'39","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":39,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:55.042511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065151180000000003,"stat_sum":{"num_bytes":12790883,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":78,"num_read_kb":73062,"num_write":53,"num_write_kb":13339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.899929+0000","last_change":"2024-04-20T02:50:28.092115+0000","last_active":"2024-04-20T02:51:08.899929+0000","last_peered":"2024-04-20T02:51:08.899929+0000","last_clean":"2024-04-20T02:51:08.899929+0000","last_became_active":"2024-04-20T02:50:28.091570+0000","last_became_peered":"2024-04-20T02:50:28.091570+0000","last_unstale":"2024-04-20T02:51:08.899929+0000","last_undegraded":"2024-04-20T02:51:08.899929+0000","last_fullsized":"2024-04-20T02:51:08.899929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:27.403210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"48'116","reported_seq":174,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.792164+0000","last_change":"2024-04-20T02:50:34.239216+0000","last_active":"2024-04-20T02:51:08.792164+0000","last_peered":"2024-04-20T02:51:08.792164+0000","last_clean":"2024-04-20T02:51:08.792164+0000","last_became_active":"2024-04-20T02:50:34.238906+0000","last_became_peered":"2024-04-20T02:50:34.238906+0000","last_unstale":"2024-04-20T02:51:08.792164+0000","last_undegraded":"2024-04-20T02:51:08.792164+0000","last_fullsized":"2024-04-20T02:51:08.792164+0000","mapping_epoch":21,"log_start":"42'113","ondisk_log_start":"42'113","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":138,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.390119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28959845,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":111,"num_read_kb":6350,"num_write":143,"num_write_kb":48210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"52'127","reported_seq":297,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.436361+0000","last_change":"2024-04-20T02:51:06.140804+0000","last_active":"2024-04-20T02:51:09.436361+0000","last_peered":"2024-04-20T02:51:09.436361+0000","last_clean":"2024-04-20T02:51:09.436361+0000","last_became_active":"2024-04-20T02:51:05.421483+0000","last_became_peered":"2024-04-20T02:51:05.421483+0000","last_unstale":"2024-04-20T02:51:09.436361+0000","last_undegraded":"2024-04-20T02:51:09.436361+0000","last_fullsized":"2024-04-20T02:51:09.436361+0000","mapping_epoch":47,"log_start":"49'124","ondisk_log_start":"49'124","created":22,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.981337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020740003999999999,"stat_sum":{"num_bytes":23383373,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":139,"num_read_kb":154428,"num_write":95,"num_write_kb":24765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":18280879,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-04-20T02:51:08.898181+0000","last_change":"2024-04-20T02:50:28.091938+0000","last_active":"2024-04-20T02:51:08.898181+0000","last_peered":"2024-04-20T02:51:08.898181+0000","last_clean":"2024-04-20T02:51:08.898181+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:51:08.898181+0000","last_undegraded":"2024-04-20T02:51:08.898181+0000","last_fullsized":"2024-04-20T02:51:08.898181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:34.990718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-20T02:51:10.445522+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:51:10.445522+0000","last_peered":"2024-04-20T02:51:10.445522+0000","last_clean":"2024-04-20T02:51:10.445522+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:51:10.445522+0000","last_undegraded":"2024-04-20T02:51:10.445522+0000","last_fullsized":"2024-04-20T02:51:10.445522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"54'67","reported_seq":185,"reported_epoch":54,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:12.452284+0000","last_change":"2024-04-20T02:51:12.450055+0000","last_active":"2024-04-20T02:51:12.452284+0000","last_peered":"2024-04-20T02:51:12.452284+0000","last_clean":"2024-04-20T02:51:12.452284+0000","last_became_active":"2024-04-20T02:50:35.413445+0000","last_became_peered":"2024-04-20T02:50:35.413445+0000","last_unstale":"2024-04-20T02:51:12.452284+0000","last_undegraded":"2024-04-20T02:51:12.452284+0000","last_fullsized":"2024-04-20T02:51:12.452284+0000","mapping_epoch":22,"log_start":"51'61","ondisk_log_start":"51'61","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":61,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:56.404298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0023455880000001,"stat_sum":{"num_bytes":10840416,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":105,"num_read_kb":85234,"num_write":44,"num_write_kb":11536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-20T02:51:11.450251+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:51:11.450251+0000","last_peered":"2024-04-20T02:51:11.450251+0000","last_clean":"2024-04-20T02:51:11.450251+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:51:11.450251+0000","last_undegraded":"2024-04-20T02:51:11.450251+0000","last_fullsized":"2024-04-20T02:51:11.450251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.432280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"52'79","reported_seq":218,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-20T02:51:09.549410+0000","last_change":"2024-04-20T02:51:06.119828+0000","last_active":"2024-04-20T02:51:09.549410+0000","last_peered":"2024-04-20T02:51:09.549410+0000","last_clean":"2024-04-20T02:51:09.549410+0000","last_became_active":"2024-04-20T02:51:04.413969+0000","last_became_peered":"2024-04-20T02:51:04.413969+0000","last_unstale":"2024-04-20T02:51:09.549410+0000","last_undegraded":"2024-04-20T02:51:09.549410+0000","last_fullsized":"2024-04-20T02:51:09.549410+0000","mapping_epoch":46,"log_start":"52'76","ondisk_log_start":"52'76","created":22,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":76,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.956090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071174699999999999,"stat_sum":{"num_bytes":14473226,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":70,"num_read_kb":59946,"num_write":70,"num_write_kb":18690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7407305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":174097947,"num_objects":82,"num_object_clones":35,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":4,"num_read":1186,"num_read_kb":1249121,"num_write":648,"num_write_kb":177176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":31203995,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":54,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":318308352,"data_stored":381035946,"data_compressed":1651560,"data_compressed_allocated":1400832,"data_compressed_original":4062327,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":48,"acting":48,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":103544715,"num_objects":127,"num_object_clones":0,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":367,"num_read_kb":17411,"num_write":501,"num_write_kb":170419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":161460224,"data_stored":175351286,"data_compressed":11184321,"data_compressed_allocated":11980800,"data_compressed_original":26456064,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509456,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67260,"kb_used_data":40168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94304580,"statfs":{"total":96636764160,"available":96567889920,"internally_reserved":0,"allocated":41132032,"data_stored":52309410,"data_compressed":16844,"data_compressed_allocated":45056,"data_compressed_original":577536,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757253,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75860,"kb_used_data":45520,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94295980,"statfs":{"total":96636764160,"available":96559083520,"internally_reserved":0,"allocated":46612480,"data_stored":52416628,"data_compressed":1083307,"data_compressed_allocated":1159168,"data_compressed_original":2473984,"omap_allocated":1657,"internal_metadata":28375431},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]}]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56988,"kb_used_data":29968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94314852,"statfs":{"total":96636764160,"available":96578408448,"internally_reserved":0,"allocated":30687232,"data_stored":36085808,"data_compressed":1041091,"data_compressed_allocated":1048576,"data_compressed_original":2203648,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91444,"kb_used_data":64424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94280396,"statfs":{"total":96636764160,"available":96543125504,"internally_reserved":0,"allocated":65970176,"data_stored":75138624,"data_compressed":1361937,"data_compressed_allocated":1327104,"data_compressed_original":3099005,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":54088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":55386112,"data_stored":63810496,"data_compressed":2040256,"data_compressed_allocated":2162688,"data_compressed_original":4795990,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.217}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98756,"kb_used_data":71728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94273084,"statfs":{"total":96636764160,"available":96535638016,"internally_reserved":0,"allocated":73449472,"data_stored":80310869,"data_compressed":3034714,"data_compressed_allocated":3219456,"data_compressed_original":7592743,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509452,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101592,"kb_used_data":70400,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94270248,"statfs":{"total":96636764160,"available":96532733952,"internally_reserved":0,"allocated":72089600,"data_stored":84378489,"data_compressed":2095144,"data_compressed_allocated":2289664,"data_compressed_original":5824512,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509454,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":124596,"kb_used_data":93480,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94247244,"statfs":{"total":96636764160,"available":96509177856,"internally_reserved":0,"allocated":95723520,"data_stored":110744185,"data_compressed":2129887,"data_compressed_allocated":2228224,"data_compressed_original":5421437,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28381184,"data_stored":30949639,"data_compressed":1843809,"data_compressed_allocated":1990656,"data_compressed_original":4657152,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11038720,"data_stored":11916936,"data_compressed":822943,"data_compressed_allocated":872448,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11038720,"data_stored":11916936,"data_compressed":822943,"data_compressed_allocated":872448,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28123136,"data_stored":30409704,"data_compressed":1915052,"data_compressed_allocated":2048000,"data_compressed_original":4419584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28381184,"data_stored":30949639,"data_compressed":1843809,"data_compressed_allocated":1990656,"data_compressed_original":4657152,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14135296,"data_stored":15322309,"data_compressed":1048539,"data_compressed_allocated":1118208,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40361984,"data_stored":43886123,"data_compressed":2887226,"data_compressed_allocated":3088384,"data_compressed_original":6758400,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44679168,"data_stored":55212658,"data_compressed":294672,"data_compressed_allocated":266240,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54341632,"data_stored":62977902,"data_compressed":538994,"data_compressed_allocated":454656,"data_compressed_original":1288573,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19058688,"data_stored":23925086,"data_compressed":218148,"data_compressed_allocated":176128,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26673152,"data_stored":33157006,"data_compressed":125204,"data_compressed_allocated":114688,"data_compressed_original":376406,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":68710400,"data_stored":81713216,"data_compressed":286078,"data_compressed_allocated":237568,"data_compressed_original":764285,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":31875072,"data_stored":36844968,"data_compressed":34768,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32444416,"data_stored":35652032,"data_compressed":142170,"data_compressed_allocated":98304,"data_compressed_original":322343,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40525824,"data_stored":51553078,"data_compressed":11526,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:51:14.677 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-20T02:51:14.677 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-04-20T02:51:14.682 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:14.716 INFO:tasks.rados.rados.0.smithi105.stdout:973: expect (ObjNum 96 snap 4 seq_num 96) 2024-04-20T02:51:14.717 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:14.716+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:14.805 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:14.806+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:14.851 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:14.852 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:14.852 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:14.852 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:14.852 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:14.862 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:14.906 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:14.906+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:14.951 INFO:tasks.rados.rados.0.smithi105.stdout:970: finishing write tid 1 to smithi10548823-48 2024-04-20T02:51:14.951 INFO:tasks.rados.rados.0.smithi105.stdout:970: finishing write tid 2 to smithi10548823-48 2024-04-20T02:51:14.951 INFO:tasks.rados.rados.0.smithi105.stdout:970: finishing write tid 3 to smithi10548823-48 2024-04-20T02:51:14.951 INFO:tasks.rados.rados.0.smithi105.stdout:970: finishing write tid 4 to smithi10548823-48 2024-04-20T02:51:14.951 INFO:tasks.rados.rados.0.smithi105.stdout:970: oid 48 updating version 0 to 55 2024-04-20T02:51:14.951 INFO:tasks.rados.rados.0.smithi105.stdout:970: oid 48 version 55 is already newer than 53 2024-04-20T02:51:14.951 INFO:tasks.rados.rados.0.smithi105.stdout:970: oid 48 version 55 is already newer than 54 2024-04-20T02:51:14.951 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 55 (ObjNum 154 snap 10 seq_num 154) dirty exists 2024-04-20T02:51:14.951 INFO:tasks.rados.rados.0.smithi105.stdout:970: left oid 48 (ObjNum 154 snap 10 seq_num 154) 2024-04-20T02:51:14.951 INFO:tasks.rados.rados.0.smithi105.stdout:974: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:15.007 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:15.007+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:15.032 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.3 46/44 [PushOp(4:cfd1e4d4:::smithi10548823-9:head, version: 45'100, data_included: [1817887~282988], data_size: 282988, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:cfd1e4d4:::smithi10548823-9:head@45'100, size: 2100875, copy_subset: [0~2100875], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2100875, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1817887, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:03.670012+0000", 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 11.361895875, 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0085824019999999994, 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670012+0000", 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670012+0000", 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670011+0000", 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670238+0000", 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022708599999999999 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670252+0000", 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3485e-05 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670264+0000", 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2007999999999999e-05 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670313+0000", 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8925000000000003e-05 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670322+0000", 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8049999999999996e-06 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.678594+0000", 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0082727180000000001 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1528 3.2 e48/47 3:7333b787:::49dfa3a36ea566ece109be00940c25f90e74796f7ebe644247ba8ecd48263ae5:head v 48'95, mlcod=0'0)", 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:15.036 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.6094724209999995, 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088605330000000003, 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423048+0000", 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00061281499999999995 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423060+0000", 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1644000000000001e-05 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423068+0000", 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7460000000000006e-06 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423108+0000", 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.977e-05 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423134+0000", 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5599e-05 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431261+0000", 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081278059999999996 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431296+0000", 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4413999999999999e-05 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(3.2 48/47 [PushOp(3:556eb17c:::3c9812f1a7ee9f26a1175ea4fab5c703e0d495f552e39ebbe4874fc5964ce285:head, version: 45'94, data_included: [0~270207], data_size: 270207, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:556eb17c:::3c9812f1a7ee9f26a1175ea4fab5c703e0d495f552e39ebbe4874fc5964ce285:head@45'94, size: 270207, copy_subset: [0~270207], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:270207, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.432273+0000", 2024-04-20T02:51:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5996350800000005, 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0067461539999999999, 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432273+0000", 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432273+0000", 2024-04-20T02:51:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432272+0000", 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432702+0000", 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00042944400000000002 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432715+0000", 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3135e-05 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432725+0000", 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0767999999999999e-05 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432765+0000", 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9779e-05 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432779+0000", 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4266e-05 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439019+0000", 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0062393290000000001 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:152 3.2 e48/47 3:54f0c00e:::2528189ec1c891f179c539fa23af2c87b9e8d2c19f70e70fac8934d3cc51ccee:head v 48'96, mlcod=0'0)", 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5990014949999996, 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020196797999999998, 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433091+0000", 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000185589 2024-04-20T02:51:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433102+0000", 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433108+0000", 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5660000000000003e-06 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439137+0000", 2024-04-20T02:51:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060284329999999997 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439166+0000", 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9278e-05 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453075+0000", 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013909163 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453103+0000", 2024-04-20T02:51:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7725999999999998e-05 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:154 3.2 e48/47 3:7d72d9ce:::7b4ada8765f224bb4a516820a83f51a402b4ee671c5f45bb3fbe5e6656cf1f14:head v 48'97, mlcod=42'91)", 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5950958629999992, 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016320239, 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436811+0000", 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437600+0000", 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078939699999999995 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437613+0000", 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2588e-05 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437624+0000", 2024-04-20T02:51:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0542e-05 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442541+0000", 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049178809999999998 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442565+0000", 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3558e-05 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453110+0000", 2024-04-20T02:51:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010545172 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453132+0000", 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1929e-05 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:144 3.2 e48/47 3:7333b787:::49dfa3a36ea566ece109be00940c25f90e74796f7ebe644247ba8ecd48263ae5:head v 48'98, mlcod=42'91)", 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.438561+0000", 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5933468160000004, 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014604997999999999, 2024-04-20T02:51:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438561+0000", 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438561+0000", 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438560+0000", 2024-04-20T02:51:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438568+0000", 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3380000000000005e-06 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438583+0000", 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4284000000000001e-05 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438591+0000", 2024-04-20T02:51:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3569999999999998e-06 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443406+0000", 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048148610000000001 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443435+0000", 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8915999999999999e-05 2024-04-20T02:51:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453135+0000", 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0097005549999999996 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453166+0000", 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0516999999999999e-05 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:155 3.2 e48/47 3:715fab8c:::b945a2927747be608263bdbe1fe814f665c73382cb3375555da5aa95a92567ff:head v 48'99, mlcod=42'91)", 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5923088080000003, 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017321785999999999, 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439598+0000", 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439954+0000", 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00035586299999999999 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439970+0000", 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5889e-05 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439981+0000", 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0754000000000001e-05 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443643+0000", 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036621259999999999 2024-04-20T02:51:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443671+0000", 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8289e-05 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456885+0000", 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013213964999999999 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456921+0000", 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:51:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1529 4.8 e48/44 4:1afa5d2b:::smithi10548823-17:head v 48'47, mlcod=0'0)", 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.455478+0000", 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5764298580000009, 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003386975, 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455478+0000", 2024-04-20T02:51:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455478+0000", 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455478+0000", 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455482+0000", 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6349999999999997e-06 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455492+0000", 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5550000000000008e-06 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455500+0000", 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9189999999999999e-06 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455600+0000", 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100413 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.455623+0000", 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3183000000000001e-05 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.458809+0000", 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003185818 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.458865+0000", 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5761999999999999e-05 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1635 3.2 e51/49 3:67667c16:::7aea211498f946c134ed12b1f5632511a2f083a0eb2114d1051f38dc7326b113:head v 51'100, mlcod=0'0)", 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 6.184417861, 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019392829, 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847489+0000", 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850819+0000", 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033292069999999998 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850831+0000", 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2320000000000001e-05 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850838+0000", 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2980000000000003e-06 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850874+0000", 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5800999999999998e-05 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850890+0000", 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5536000000000002e-05 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866849+0000", 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015959346999999999 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866883+0000", 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3776999999999999e-05 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.5832696219999995, 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:51:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1664 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'77, mlcod=50'73)", 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.5221906389999997, 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012476671999999999, 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509983+0000", 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00026571599999999999 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509992+0000", 2024-04-20T02:51:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509999+0000", 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1450000000000002e-06 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.510050+0000", 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0562000000000001e-05 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.510064+0000", 2024-04-20T02:51:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4586000000000001e-05 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.522152+0000", 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012087736 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.522194+0000", 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1851999999999998e-05 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1665 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'78, mlcod=50'73)", 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.5086697689999999, 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025918594, 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523236+0000", 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523658+0000", 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000422082 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523676+0000", 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7716000000000001e-05 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523686+0000", 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9790000000000005e-06 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523734+0000", 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7579e-05 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523759+0000", 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5593e-05 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549122+0000", 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025363087999999999 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549157+0000", 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4242999999999997e-05 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.5767613899999997, 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034212370999999998, 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456260+0000", 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011140480000000001 2024-04-20T02:51:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456273+0000", 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456280+0000", 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1130000000000003e-06 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456331+0000", 2024-04-20T02:51:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1446999999999997e-05 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456357+0000", 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6001999999999999e-05 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489309+0000", 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032951972000000003 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489359+0000", 2024-04-20T02:51:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9552000000000003e-05 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1676 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'129, mlcod=49'126)", 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.475345+0000", 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.5565627390000003, 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014576344999999999, 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.475345+0000", 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.475345+0000", 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.475344+0000", 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.476165+0000", 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00082085800000000005 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.476178+0000", 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3426e-05 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.476187+0000", 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4330000000000004e-06 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.476720+0000", 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053326399999999998 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.476743+0000", 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2821999999999999e-05 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489890+0000", 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013146893 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489921+0000", 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1590000000000001e-05 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.0:330 4.3 e53/47 4:cd374330:::smithi10548823-1:f v 53'136, mlcod=53'130)", 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.492384+0000", 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.5395237709999998, 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0032585980000000001, 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.492384+0000", 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.492384+0000", 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.492383+0000", 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.492391+0000", 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5789999999999998e-06 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.492399+0000", 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.541e-06 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.492407+0000", 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8420000000000005e-06 2024-04-20T02:51:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.492466+0000", 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8307999999999998e-05 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.492486+0000", 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0128000000000001e-05 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.495605+0000", 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0031190219999999999 2024-04-20T02:51:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.495643+0000", 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7817000000000001e-05 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.0:332 4.3 e53/47 4:cfd1e4d4:::smithi10548823-9:d v 53'140, mlcod=53'134)", 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.496760+0000", 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.5351481759999999, 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0040761319999999997, 2024-04-20T02:51:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.496760+0000", 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.496760+0000", 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.496759+0000", 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.496765+0000", 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3709999999999996e-06 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.496781+0000", 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5307000000000001e-05 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.496786+0000", 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0490000000000003e-06 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.496818+0000", 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1980999999999998e-05 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.496838+0000", 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9862999999999999e-05 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.500812+0000", 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0039745179999999998 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.500836+0000", 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3509000000000001e-05 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1775 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'41, mlcod=53'38)", 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:15.100 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.2783799, 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038672274999999999, 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754868+0000", 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013404619999999999 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754881+0000", 2024-04-20T02:51:15.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.323e-05 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754892+0000", 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1154e-05 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754967+0000", 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4326999999999995e-05 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.755006+0000", 2024-04-20T02:51:15.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9338000000000002e-05 2024-04-20T02:51:15.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792154+0000", 2024-04-20T02:51:15.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037147924999999998 2024-04-20T02:51:15.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792200+0000", 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6044999999999997e-05 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1776 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'42, mlcod=53'38)", 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.2583997220000001, 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019186657999999999, 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773507+0000", 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774271+0000", 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076432799999999999 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774284+0000", 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2483000000000001e-05 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774294+0000", 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6569999999999992e-06 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774338+0000", 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4576000000000001e-05 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774358+0000", 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9449999999999998e-05 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792664+0000", 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018306413000000001 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792695+0000", 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0685000000000002e-05 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1822 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'94, mlcod=55'92)", 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.92835776699999994, 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031823907999999998, 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:15.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103786+0000", 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000236658 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103797+0000", 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0951000000000001e-05 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103804+0000", 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0960000000000003e-06 2024-04-20T02:51:15.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103860+0000", 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5781e-05 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103887+0000", 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6999999999999999e-05 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135333+0000", 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031446004 2024-04-20T02:51:15.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135374+0000", 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0778999999999997e-05 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1823 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'95, mlcod=55'92)", 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.91387076099999998, 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017980995999999999, 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:15.113 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118036+0000", 2024-04-20T02:51:15.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118363+0000", 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00032734600000000001 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118381+0000", 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7816e-05 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118391+0000", 2024-04-20T02:51:15.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0155e-05 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119591+0000", 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011995560000000001 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119618+0000", 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7475e-05 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135988+0000", 2024-04-20T02:51:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016370096000000001 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.136018+0000", 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9745999999999999e-05 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:15.117 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:15.119 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:15.120 INFO:tasks.rados.rados.0.smithi105.stdout:972: finishing write tid 1 to smithi10548823-2 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:972: finishing write tid 2 to smithi10548823-2 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:972: finishing write tid 3 to smithi10548823-2 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:972: finishing write tid 4 to smithi10548823-2 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:972: oid 2 updating version 0 to 87 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:972: oid 2 version 87 is already newer than 86 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:972: oid 2 updating version 87 to 88 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 88 (ObjNum 155 snap 10 seq_num 155) dirty exists 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:972: left oid 2 (ObjNum 155 snap 10 seq_num 155) 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:977: finishing write tid 1 to smithi10548823-10 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:977: finishing write tid 2 to smithi10548823-10 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:977: finishing write tid 3 to smithi10548823-10 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:977: finishing write tid 4 to smithi10548823-10 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:977: oid 10 updating version 0 to 96 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:977: oid 10 version 96 is already newer than 95 2024-04-20T02:51:15.121 INFO:tasks.rados.rados.0.smithi105.stdout:977: oid 10 version 96 is already newer than 94 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 96 (ObjNum 156 snap 10 seq_num 156) dirty exists 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:977: left oid 10 (ObjNum 156 snap 10 seq_num 156) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:966: done (12 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:967: done (11 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:969: done (10 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:970: done (9 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:971: done (8 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:972: done (7 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:973: done (6 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:974: done (5 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:975: done (4 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:976: done (3 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:977: done (2 left) 2024-04-20T02:51:15.122 INFO:tasks.rados.rados.0.smithi105.stdout:978: done (1 left) 2024-04-20T02:51:15.123 INFO:tasks.rados.rados.0.smithi105.stdout:979: done (0 left) 2024-04-20T02:51:15.123 INFO:tasks.rados.rados.0.smithi105.stdout:980: rollback oid 16 current snap is 11 2024-04-20T02:51:15.123 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 16 to 6 2024-04-20T02:51:15.123 INFO:tasks.rados.rados.0.smithi105.stdout:981: write oid 15 current snap is 11 2024-04-20T02:51:15.123 INFO:tasks.rados.rados.0.smithi105.stdout:981: seq_num 157 ranges {638699=541153,1770967=531722} 2024-04-20T02:51:15.123 INFO:tasks.rados.rados.0.smithi105.stdout:981: writing smithi10548823-15 from 638699 to 1179852 tid 1 2024-04-20T02:51:15.123 INFO:tasks.rados.rados.0.smithi105.stdout:981: writing smithi10548823-15 from 1770967 to 2302689 tid 2 2024-04-20T02:51:15.123 INFO:tasks.rados.rados.0.smithi105.stdout:982: snap_create 2024-04-20T02:51:15.123 INFO:tasks.rados.rados.0.smithi105.stdout:980: finishing rollback tid 1 to smithi10548823-16 2024-04-20T02:51:15.123 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 0 (ObjNum 188079136 snap 32641 seq_num 304) dirty dne 2024-04-20T02:51:15.123 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:15.118+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:15.126 INFO:tasks.rados.rados.0.smithi105.stdout:981: finishing write tid 1 to smithi10548823-15 2024-04-20T02:51:15.129 INFO:tasks.rados.rados.0.smithi105.stdout:981: finishing write tid 2 to smithi10548823-15 2024-04-20T02:51:15.129 INFO:tasks.rados.rados.0.smithi105.stdout:981: finishing write tid 3 to smithi10548823-15 2024-04-20T02:51:15.129 INFO:tasks.rados.rados.0.smithi105.stdout:981: finishing write tid 4 to smithi10548823-15 2024-04-20T02:51:15.129 INFO:tasks.rados.rados.0.smithi105.stdout:981: oid 15 updating version 0 to 144 2024-04-20T02:51:15.130 INFO:tasks.rados.rados.0.smithi105.stdout:981: oid 15 updating version 144 to 145 2024-04-20T02:51:15.130 INFO:tasks.rados.rados.0.smithi105.stdout:981: oid 15 version 145 is already newer than 143 2024-04-20T02:51:15.130 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 145 (ObjNum 157 snap 11 seq_num 157) dirty exists 2024-04-20T02:51:15.130 INFO:tasks.rados.rados.0.smithi105.stdout:981: left oid 15 (ObjNum 157 snap 11 seq_num 157) 2024-04-20T02:51:15.219 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:15.220+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:15.253 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:15.253 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:15.263 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:15.347 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:15.347+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:15.389 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:15.421 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:15.421+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:15.433 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:15.434 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:15.445 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:15.473 INFO:tasks.rados.rados.0.smithi105.stdout:980: done (2 left) 2024-04-20T02:51:15.474 INFO:tasks.rados.rados.0.smithi105.stdout:981: done (1 left) 2024-04-20T02:51:15.474 INFO:tasks.rados.rados.0.smithi105.stdout:982: done (0 left) 2024-04-20T02:51:15.474 INFO:tasks.rados.rados.0.smithi105.stdout:983: write (excl) oid 28 current snap is 12 2024-04-20T02:51:15.474 INFO:tasks.rados.rados.0.smithi105.stdout:983: seq_num 158 ranges {655506=616402,1829449=675380} 2024-04-20T02:51:15.475 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:51:15.476 INFO:teuthology.orchestra.run.smithi105.stderr:set pool 4 pg_num to 26 2024-04-20T02:51:15.481 INFO:tasks.rados.rados.0.smithi105.stdout:983: writing smithi10548823-28 from 655506 to 1271908 tid 1 2024-04-20T02:51:15.489 INFO:tasks.rados.rados.0.smithi105.stdout:983: writing smithi10548823-28 from 1829449 to 2504829 tid 2 2024-04-20T02:51:15.489 INFO:tasks.rados.rados.0.smithi105.stdout:984: read oid 48 snap 6 2024-04-20T02:51:15.489 INFO:tasks.rados.rados.0.smithi105.stdout:984: expect (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:51:15.489 INFO:tasks.rados.rados.0.smithi105.stdout:985: snap_create 2024-04-20T02:51:15.495 INFO:tasks.rados.rados.0.smithi105.stdout:984: expect (ObjNum 82 snap 1 seq_num 82) 2024-04-20T02:51:15.539 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:15.539+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:15.549 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:15.570 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:15.590 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:15.590 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:15.590 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:15.602 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:15.611 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:15.623 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:15.623+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:15.633 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:15.653 INFO:tasks.rados.rados.0.smithi105.stdout:983: finishing write tid 1 to smithi10548823-28 2024-04-20T02:51:15.653 INFO:tasks.rados.rados.0.smithi105.stdout:983: finishing write tid 2 to smithi10548823-28 2024-04-20T02:51:15.653 INFO:tasks.rados.rados.0.smithi105.stdout:983: finishing write tid 3 to smithi10548823-28 2024-04-20T02:51:15.653 INFO:tasks.rados.rados.0.smithi105.stdout:983: finishing write tid 4 to smithi10548823-28 2024-04-20T02:51:15.653 INFO:tasks.rados.rados.0.smithi105.stdout:983: oid 28 updating version 0 to 99 2024-04-20T02:51:15.653 INFO:tasks.rados.rados.0.smithi105.stdout:983: oid 28 updating version 99 to 101 2024-04-20T02:51:15.653 INFO:tasks.rados.rados.0.smithi105.stdout:983: oid 28 version 101 is already newer than 100 2024-04-20T02:51:15.654 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 101 (ObjNum 158 snap 12 seq_num 158) dirty exists 2024-04-20T02:51:15.654 INFO:tasks.rados.rados.0.smithi105.stdout:983: left oid 28 (ObjNum 158 snap 12 seq_num 158) 2024-04-20T02:51:15.654 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:15.673 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:15.692 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:15.693 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:15.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:15.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:15.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:15.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:15.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:15.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:15.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:15.724 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:15.725+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:15.746 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:15.746 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:15.756 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:15.825 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:15.825+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:15.885 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:15.885 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:15.895 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:15.939 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:15.939+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:16.024 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:16.024 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:16.027 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:16.028+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:16.034 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:16.128 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:16.128+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:16.203 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:16.204 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:16.214 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:16.228 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:16.229+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:16.329 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:16.329+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:16.341 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:16.341 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:16.352 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:16.430 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:16.431+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:16.482 INFO:tasks.rados.rados.0.smithi105.stdout:983: done (2 left) 2024-04-20T02:51:16.482 INFO:tasks.rados.rados.0.smithi105.stdout:984: done (1 left) 2024-04-20T02:51:16.482 INFO:tasks.rados.rados.0.smithi105.stdout:985: done (0 left) 2024-04-20T02:51:16.482 INFO:tasks.rados.rados.0.smithi105.stdout:986: delete oid 3 current snap is 13 2024-04-20T02:51:16.485 INFO:tasks.rados.rados.0.smithi105.stdout:986: done (0 left) 2024-04-20T02:51:16.485 INFO:tasks.rados.rados.0.smithi105.stdout:987: rollback oid 15 current snap is 13 2024-04-20T02:51:16.485 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 15 to 9 2024-04-20T02:51:16.485 INFO:tasks.rados.rados.0.smithi105.stdout:988: tier_promote oid 23 2024-04-20T02:51:16.489 INFO:tasks.rados.rados.0.smithi105.stdout:989: write (excl) oid 40 current snap is 13 2024-04-20T02:51:16.489 INFO:tasks.rados.rados.0.smithi105.stdout:989: seq_num 160 ranges {689120=483253,1946413=679049} 2024-04-20T02:51:16.489 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:51:16.492 INFO:tasks.rados.rados.0.smithi105.stdout:989: writing smithi10548823-40 from 689120 to 1172373 tid 1 2024-04-20T02:51:16.501 INFO:tasks.rados.rados.0.smithi105.stdout:989: writing smithi10548823-40 from 1946413 to 2625462 tid 2 2024-04-20T02:51:16.501 INFO:tasks.rados.rados.0.smithi105.stdout:990: read oid 39 snap -1 2024-04-20T02:51:16.501 INFO:tasks.rados.rados.0.smithi105.stdout:990: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:16.501 INFO:tasks.rados.rados.0.smithi105.stdout:991: set_chunk oid 27 target oid 2024-04-20T02:51:16.531 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:16.531+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:16.551 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:16.551 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:16.566 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:16.566 INFO:tasks.rados.rados.0.smithi105.stdout:991: set_chunk oid 27 offset: 796672 length: 2496000 target oid 3feaa61f599eda94329d7d43d5ba69924479334596b8ea519ae4d22fde641e76 offset: 0 snap -1 2024-04-20T02:51:16.566 INFO:tasks.rados.rados.0.smithi105.stdout:992: set_chunk oid 25 target oid 2024-04-20T02:51:16.617 INFO:tasks.rados.rados.0.smithi105.stdout:992: set_chunk oid 25 offset: 861184 length: 1961472 target oid 2b25199035ded10a93199db7b3349081891a7285cb22373a9eb6d8556fd77165 offset: 0 snap -1 2024-04-20T02:51:16.617 INFO:tasks.rados.rados.0.smithi105.stdout:993: read oid 49 snap -1 2024-04-20T02:51:16.617 INFO:tasks.rados.rados.0.smithi105.stdout:993: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:16.617 INFO:tasks.rados.rados.0.smithi105.stdout:994: snap_remove snap 10 2024-04-20T02:51:16.631 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:16.632+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:16.698 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:16.698 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:16.709 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:16.750 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:16.750+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:16.833 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:16.834+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:16.877 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:16.877 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:16.887 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:16.972 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:16.972+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:17.035 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:17.035+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:17.055 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:17.055 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:17.066 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:17.136 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:17.136+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:17.205 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:17.205 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:17.215 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:17.258 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:17.259+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:17.338 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:17.338+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:17.343 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:17.343 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:17.354 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:17.439 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:17.440+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:17.494 INFO:tasks.rados.rados.0.smithi105.stdout:994: done (7 left) 2024-04-20T02:51:17.494 INFO:tasks.rados.rados.0.smithi105.stdout:995: read oid 1 snap -1 2024-04-20T02:51:17.494 INFO:tasks.rados.rados.0.smithi105.stdout:995: expect (ObjNum 133 snap 9 seq_num 133) 2024-04-20T02:51:17.494 INFO:tasks.rados.rados.0.smithi105.stdout:996: read oid 24 snap 9 2024-04-20T02:51:17.494 INFO:tasks.rados.rados.0.smithi105.stdout:996: expect (ObjNum 132 snap 9 seq_num 132) 2024-04-20T02:51:17.494 INFO:tasks.rados.rados.0.smithi105.stdout:997: read oid 7 snap -1 2024-04-20T02:51:17.494 INFO:tasks.rados.rados.0.smithi105.stdout:997: expect (ObjNum 105 snap 5 seq_num 105) 2024-04-20T02:51:17.494 INFO:tasks.rados.rados.0.smithi105.stdout:998: read oid 43 snap -1 2024-04-20T02:51:17.494 INFO:tasks.rados.rados.0.smithi105.stdout:998: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:51:17.495 INFO:tasks.rados.rados.0.smithi105.stdout:999: read oid 13 snap -1 2024-04-20T02:51:17.495 INFO:tasks.rados.rados.0.smithi105.stdout:999: expect (ObjNum 151 snap 10 seq_num 151) 2024-04-20T02:51:17.495 INFO:tasks.rados.rados.0.smithi105.stdout:987: finishing rollback tid 0 to smithi10548823-15 2024-04-20T02:51:17.495 INFO:tasks.rados.rados.0.smithi105.stdout:987: finishing rollback tid 1 to smithi10548823-15 2024-04-20T02:51:17.495 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 150 (ObjNum 33 snap 0 seq_num 61320352) dirty dne 2024-04-20T02:51:17.495 INFO:tasks.rados.rados.0.smithi105.stdout:988: got (0) Success 2024-04-20T02:51:17.495 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 37 (ObjNum 130 snap 9 seq_num 130) dirty exists 2024-04-20T02:51:17.495 INFO:tasks.rados.rados.0.smithi105.stdout:990: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:17.532 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:17.532 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:17.540 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:17.540+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:17.543 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:989: finishing write tid 1 to smithi10548823-40 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:989: finishing write tid 2 to smithi10548823-40 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:989: finishing write tid 3 to smithi10548823-40 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:989: finishing write tid 4 to smithi10548823-40 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:989: oid 40 updating version 0 to 153 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:989: oid 40 version 153 is already newer than 151 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:989: oid 40 version 153 is already newer than 152 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 153 (ObjNum 160 snap 13 seq_num 160) dirty exists 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:989: left oid 40 (ObjNum 160 snap 13 seq_num 160) 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:991: finishing set_chunk to oid 27 2024-04-20T02:51:17.564 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 796672 length: 2496000 2024-04-20T02:51:17.565 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 103 snap 5 seq_num 103) dirty exists 2024-04-20T02:51:17.565 INFO:tasks.rados.rados.0.smithi105.stdout:992: finishing set_chunk to oid 25 2024-04-20T02:51:17.565 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 861184 length: 1961472 2024-04-20T02:51:17.565 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 70 snap 0 seq_num 70) dirty exists 2024-04-20T02:51:17.565 INFO:tasks.rados.rados.0.smithi105.stdout:1000: read oid 50 snap -1 2024-04-20T02:51:17.566 INFO:tasks.rados.rados.0.smithi105.stdout:1000: expect (ObjNum 146 snap 10 seq_num 146) 2024-04-20T02:51:17.566 INFO:tasks.rados.rados.0.smithi105.stdout:987: done (12 left) 2024-04-20T02:51:17.566 INFO:tasks.rados.rados.0.smithi105.stdout:988: done (11 left) 2024-04-20T02:51:17.566 INFO:tasks.rados.rados.0.smithi105.stdout:989: done (10 left) 2024-04-20T02:51:17.566 INFO:tasks.rados.rados.0.smithi105.stdout:990: done (9 left) 2024-04-20T02:51:17.566 INFO:tasks.rados.rados.0.smithi105.stdout:991: done (8 left) 2024-04-20T02:51:17.566 INFO:tasks.rados.rados.0.smithi105.stdout:992: done (7 left) 2024-04-20T02:51:17.566 INFO:tasks.rados.rados.0.smithi105.stdout:1001: tier_promote oid 17 2024-04-20T02:51:17.566 INFO:tasks.rados.rados.0.smithi105.stdout:993: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:17.642 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:17.642+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:17.702 INFO:tasks.rados.rados.0.smithi105.stdout:1000: expect (ObjNum 146 snap 10 seq_num 146) 2024-04-20T02:51:17.706 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:17.712 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:17.713 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:17.723 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:17.764 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:17.765+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:17.817 INFO:tasks.rados.rados.0.smithi105.stdout:993: done (7 left) 2024-04-20T02:51:17.817 INFO:tasks.rados.rados.0.smithi105.stdout:1000: done (6 left) 2024-04-20T02:51:17.817 INFO:tasks.rados.rados.0.smithi105.stdout:1002: read oid 42 snap -1 2024-04-20T02:51:17.817 INFO:tasks.rados.rados.0.smithi105.stdout:1002: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:51:17.817 INFO:tasks.rados.rados.0.smithi105.stdout:1003: write oid 40 current snap is 13 2024-04-20T02:51:17.817 INFO:tasks.rados.rados.0.smithi105.stdout:1003: seq_num 161 ranges {705927=558502,2004895=422707} 2024-04-20T02:51:17.824 INFO:tasks.rados.rados.0.smithi105.stdout:1003: writing smithi10548823-40 from 705927 to 1264429 tid 1 2024-04-20T02:51:17.830 INFO:tasks.rados.rados.0.smithi105.stdout:1003: writing smithi10548823-40 from 2004895 to 2427602 tid 2 2024-04-20T02:51:17.830 INFO:tasks.rados.rados.0.smithi105.stdout:1004: write (excl) oid 30 current snap is 13 2024-04-20T02:51:17.831 INFO:tasks.rados.rados.0.smithi105.stdout:1004: seq_num 162 ranges {722734=633751,2063377=659357} 2024-04-20T02:51:17.838 INFO:tasks.rados.rados.0.smithi105.stdout:1004: writing smithi10548823-30 from 722734 to 1356485 tid 1 2024-04-20T02:51:17.843 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:17.844+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:17.846 INFO:tasks.rados.rados.0.smithi105.stdout:1004: writing smithi10548823-30 from 2063377 to 2722734 tid 2 2024-04-20T02:51:17.846 INFO:tasks.rados.rados.0.smithi105.stdout:1005: snap_create 2024-04-20T02:51:17.855 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:17.874 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:17.892 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:17.892 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:17.893 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:17.903 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:17.910 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:17.930 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:17.944 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:17.945+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:17.949 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:17.967 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:17.987 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:17.988 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:17.988 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:17.988 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:17.988 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:17.988 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:17.988 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:17.988 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:17.988 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:18.045 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:18.046+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:18.074 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:18.074 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:18.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:18.074 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:18.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.074 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.074 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:18.074 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.337836621000001, 2024-04-20T02:51:18.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:51:18.074 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:51:18.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:51:18.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.080 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.081 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.081 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:18.081 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.306547559999998, 2024-04-20T02:51:18.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:51:18.081 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.081 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.081 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:51:18.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:51:18.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:51:18.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:51:18.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.301857104, 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:51:18.087 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.295100634999997, 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:51:18.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:51:18.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:51:18.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:51:18.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.285285123999998, 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.099 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:51:18.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:51:18.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.284241192000003, 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.105 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.111 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.111 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:18.111 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.284220032, 2024-04-20T02:51:18.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:51:18.111 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.111 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:18.111 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.111 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.111 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.115 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.076649451000002, 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:51:18.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:51:18.119 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.062454805999998, 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:18.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:51:18.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:51:18.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:51:18.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.056487549000003, 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:51:18.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:51:18.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.048081545999999, 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.944449505000001, 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.132 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:51:18.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.713853194999999, 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.136 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:51:18.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:51:18.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.686909045999997, 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:51:18.140 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:18.144 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.661550779000002, 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:51:18.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:51:18.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.633410335000001, 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.152 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.596041730000003, 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:51:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:51:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:51:18.156 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.595744807000003, 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:18.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:51:18.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:51:18.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.590661689000001, 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.563659129999998, 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:18.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:51:18.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:51:18.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.169 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:18.171 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:18.171 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:18.171+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:18.271 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:18.272+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:18.341 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:18.341 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:18.341 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:18.341 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:18.351 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:18.372 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:18.372+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:18.473 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:18.473+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:18.495 INFO:tasks.rados.rados.0.smithi105.stdout:1005: done (9 left) 2024-04-20T02:51:18.495 INFO:tasks.rados.rados.0.smithi105.stdout:1006: read oid 39 snap -1 2024-04-20T02:51:18.495 INFO:tasks.rados.rados.0.smithi105.stdout:1006: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:18.495 INFO:tasks.rados.rados.0.smithi105.stdout:1007: read oid 18 snap -1 2024-04-20T02:51:18.495 INFO:tasks.rados.rados.0.smithi105.stdout:1007: expect (ObjNum 83 snap 1 seq_num 83) 2024-04-20T02:51:18.495 INFO:tasks.rados.rados.0.smithi105.stdout:1008: set_chunk oid 11 target oid 2024-04-20T02:51:18.495 INFO:tasks.rados.rados.0.smithi105.stdout:1008: set_chunk oid 11 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:51:18.495 INFO:tasks.rados.rados.0.smithi105.stdout:1009: read oid 8 snap 9 2024-04-20T02:51:18.495 INFO:tasks.rados.rados.0.smithi105.stdout:1009: expect (ObjNum 86 snap 2 seq_num 86) 2024-04-20T02:51:18.495 INFO:tasks.rados.rados.0.smithi105.stdout:1010: read oid 35 snap 11 2024-04-20T02:51:18.496 INFO:tasks.rados.rados.0.smithi105.stdout:1010: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:51:18.496 INFO:tasks.rados.rados.0.smithi105.stdout:1011: read oid 48 snap -1 2024-04-20T02:51:18.496 INFO:tasks.rados.rados.0.smithi105.stdout:1011: expect (ObjNum 154 snap 10 seq_num 154) 2024-04-20T02:51:18.496 INFO:tasks.rados.rados.0.smithi105.stdout:1012: rollback oid 41 current snap is 14 2024-04-20T02:51:18.496 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 41 to 12 2024-04-20T02:51:18.496 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:18.502 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:51:18.514 INFO:tasks.rados.rados.0.smithi105.stdout:1010: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:51:18.537 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:18.537 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:18.537 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:18.537 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:18.538 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:18.550 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:18.574 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:18.574+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:18.592 INFO:tasks.rados.rados.0.smithi105.stdout:998: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:51:18.675 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:18.675+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:18.685 INFO:tasks.rados.rados.0.smithi105.stdout:997: expect (ObjNum 105 snap 5 seq_num 105) 2024-04-20T02:51:18.722 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:18.722 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:18.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:18.722 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:18.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 59.117333635000001, 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:18.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:18.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:51:18.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:18.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:51:18.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:51:18.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:51:18.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:51:18.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:51:18.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:51:18.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:51:18.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:51:18.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:18.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:51:18.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:51:18.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:51:18.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:51:18.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:51:18.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:51:18.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:18.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:51:18.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:51:18.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:51:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:51:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:51:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:51:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:51:18.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:51:18.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:51:18.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:51:18.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:51:18.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:51:18.732 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.733 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.733 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.733 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:30 4.8 4:1c68e542:::smithi10548823-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.733 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:18.733 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.357181447000002, 2024-04-20T02:51:18.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033039351000000002, 2024-04-20T02:51:18.734 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.734 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.734 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.734 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.734 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.735 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 30 2024-04-20T02:51:18.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.735 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:18.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364066+0000", 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364312+0000", 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000246708 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364328+0000", 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5376999999999999e-05 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364339+0000", 2024-04-20T02:51:18.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.092e-05 2024-04-20T02:51:18.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375692+0000", 2024-04-20T02:51:18.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011353103999999999 2024-04-20T02:51:18.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375805+0000", 2024-04-20T02:51:18.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000113414 2024-04-20T02:51:18.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:18.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375970+0000", 2024-04-20T02:51:18.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016508399999999999 2024-04-20T02:51:18.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388615+0000", 2024-04-20T02:51:18.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012645145 2024-04-20T02:51:18.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391156+0000", 2024-04-20T02:51:18.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025406410000000002 2024-04-20T02:51:18.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396990+0000", 2024-04-20T02:51:18.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0058340390000000001 2024-04-20T02:51:18.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397042+0000", 2024-04-20T02:51:18.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2454000000000003e-05 2024-04-20T02:51:18.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397106+0000", 2024-04-20T02:51:18.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3632000000000003e-05 2024-04-20T02:51:18.742 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.742 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.742 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.743 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:31 4.8 4:1c68e542:::smithi10548823-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.743 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:18.743 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.356894750999999, 2024-04-20T02:51:18.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032910542000000001, 2024-04-20T02:51:18.743 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.743 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.743 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.743 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.744 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.744 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 31 2024-04-20T02:51:18.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.744 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:18.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:18.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:18.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364357+0000", 2024-04-20T02:51:18.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1160000000000001e-06 2024-04-20T02:51:18.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364363+0000", 2024-04-20T02:51:18.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1680000000000001e-06 2024-04-20T02:51:18.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364367+0000", 2024-04-20T02:51:18.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8750000000000002e-06 2024-04-20T02:51:18.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388814+0000", 2024-04-20T02:51:18.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024447437999999998 2024-04-20T02:51:18.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388914+0000", 2024-04-20T02:51:18.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9557999999999998e-05 2024-04-20T02:51:18.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:18.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389062+0000", 2024-04-20T02:51:18.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014799600000000001 2024-04-20T02:51:18.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391160+0000", 2024-04-20T02:51:18.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020979760000000001 2024-04-20T02:51:18.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396038+0000", 2024-04-20T02:51:18.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048784689999999999 2024-04-20T02:51:18.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397155+0000", 2024-04-20T02:51:18.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001116286 2024-04-20T02:51:18.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397208+0000", 2024-04-20T02:51:18.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3273000000000003e-05 2024-04-20T02:51:18.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397264+0000", 2024-04-20T02:51:18.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5962999999999999e-05 2024-04-20T02:51:18.752 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.753 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.753 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.753 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:32 4.8 4:1c68e542:::smithi10548823-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.753 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:18.753 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.356876036999999, 2024-04-20T02:51:18.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033121840999999999, 2024-04-20T02:51:18.754 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.754 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:18.754 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.754 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.754 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.754 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 32 2024-04-20T02:51:18.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.755 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:18.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:18.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:18.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364375+0000", 2024-04-20T02:51:18.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1149999999999998e-06 2024-04-20T02:51:18.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364382+0000", 2024-04-20T02:51:18.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4259999999999997e-06 2024-04-20T02:51:18.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364387+0000", 2024-04-20T02:51:18.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6129999999999997e-06 2024-04-20T02:51:18.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389813+0000", 2024-04-20T02:51:18.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025426621 2024-04-20T02:51:18.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:18.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389905+0000", 2024-04-20T02:51:18.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2011999999999999e-05 2024-04-20T02:51:18.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397299+0000", 2024-04-20T02:51:18.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007393897 2024-04-20T02:51:18.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397395+0000", 2024-04-20T02:51:18.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5293999999999999e-05 2024-04-20T02:51:18.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397494+0000", 2024-04-20T02:51:18.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9289000000000004e-05 2024-04-20T02:51:18.764 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.764 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.764 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.765 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:51:18.765 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:18.765 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.353467236, 2024-04-20T02:51:18.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:51:18.765 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.766 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.766 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:18.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:18.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:51:18.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:51:18.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:51:18.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:51:18.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:51:18.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:51:18.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:51:18.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:51:18.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:51:18.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:51:18.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:51:18.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.772 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.240164194999998, 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:51:18.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:51:18.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:51:18.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:51:18.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:51:18.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:51:18.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:51:18.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:51:18.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:51:18.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:51:18.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:51:18.779 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.779 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.779 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.779 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:66 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.779 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:18.779 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.234806306999999, 2024-04-20T02:51:18.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033365526, 2024-04-20T02:51:18.780 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.780 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.780 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.780 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.780 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.780 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 66 2024-04-20T02:51:18.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.780 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486438+0000", 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486778+0000", 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000339709 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486798+0000", 2024-04-20T02:51:18.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0462000000000002e-05 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486812+0000", 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3589e-05 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496889+0000", 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010077477 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496978+0000", 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8205e-05 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.497125+0000", 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000147665 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518444+0000", 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.021318713 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519407+0000", 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00096312999999999998 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519727+0000", 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031953500000000002 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519764+0000", 2024-04-20T02:51:18.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7406000000000001e-05 2024-04-20T02:51:18.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519807+0000", 2024-04-20T02:51:18.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3402000000000002e-05 2024-04-20T02:51:18.786 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.786 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:67 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2268912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.234421122999997, 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033307638000000001, 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.787 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 67 2024-04-20T02:51:18.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.788 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:18.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486826+0000", 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486831+0000", 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0189999999999997e-06 2024-04-20T02:51:18.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486839+0000", 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4710000000000001e-06 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486844+0000", 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.57e-06 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509240+0000", 2024-04-20T02:51:18.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022395241 2024-04-20T02:51:18.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509338+0000", 2024-04-20T02:51:18.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8034000000000004e-05 2024-04-20T02:51:18.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509498+0000", 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015993999999999999 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518471+0000", 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0089732730000000004 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519859+0000", 2024-04-20T02:51:18.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013877130000000001 2024-04-20T02:51:18.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520042+0000", 2024-04-20T02:51:18.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182981 2024-04-20T02:51:18.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520084+0000", 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2518999999999997e-05 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520135+0000", 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0491000000000002e-05 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.234398677999998, 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.795 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:51:18.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:51:18.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:161 4.3 e29/27 4:c32272c0:::smithi10548823-41:head v 29'16, mlcod=29'14)", 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:18.800 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.886962586999999, 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031990785000000001, 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834286+0000", 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.834285+0000", 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835168+0000", 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00088287800000000003 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835180+0000", 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2340999999999999e-05 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835187+0000", 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7830000000000001e-06 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835227+0000", 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0754000000000003e-05 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835243+0000", 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5607000000000002e-05 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866234+0000", 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030990891 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.866276+0000", 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2469999999999998e-05 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.880378542999999, 2024-04-20T02:51:18.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:18.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:51:18.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.870014937999997, 2024-04-20T02:51:18.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:51:18.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.861930459, 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:51:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:51:18.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:51:18.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:981 4.7 e32/27 4:e9876a63:::smithi10548823-30:head v 32'19, mlcod=30'17)", 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.131877985000003, 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033092985999999998, 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.820 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589369+0000", 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590134+0000", 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076479 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590148+0000", 2024-04-20T02:51:18.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3679999999999999e-05 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590157+0000", 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590209+0000", 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2237999999999997e-05 2024-04-20T02:51:18.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590235+0000", 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6251e-05 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622430+0000", 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032194953999999998 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622463+0000", 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2780000000000001e-05 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1230 4.5 e38/22 4:a94dc531:::smithi10548823-32:head v 38'11, mlcod=29'8)", 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.308732587000002, 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033312785999999997, 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.825 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412515+0000", 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412827+0000", 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031200999999999999 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412835+0000", 2024-04-20T02:51:18.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.61e-06 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412842+0000", 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6990000000000003e-06 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412921+0000", 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8380000000000005e-05 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412944+0000", 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3609e-05 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445795+0000", 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032850279000000003 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445828+0000", 2024-04-20T02:51:18.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3673000000000001e-05 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.580372233999999, 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.830 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:51:18.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:18.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:51:18.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.575235123999999, 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.837 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:51:18.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:18.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:51:18.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:51:18.845 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.845 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.845 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.845 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.574962112000001, 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.846 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:51:18.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:51:18.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:51:18.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.852 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.574932727, 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:18.853 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:18.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:51:18.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:51:18.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:51:18.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:18.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:51:18.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:51:18.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:51:18.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:51:18.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:51:18.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:51:18.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:51:18.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:51:18.859 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.859 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.2660387620000009, 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035060916999999997, 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456603+0000", 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001394295 2024-04-20T02:51:18.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456616+0000", 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2605999999999999e-05 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456631+0000", 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4735e-05 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456682+0000", 2024-04-20T02:51:18.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1048000000000003e-05 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456705+0000", 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2881e-05 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490235+0000", 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033530250999999997 2024-04-20T02:51:18.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:18.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:18.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:18.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490270+0000", 2024-04-20T02:51:18.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5379000000000001e-05 2024-04-20T02:51:18.864 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.864 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.864 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.864 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:18.864 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:18.864 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:18.866 INFO:tasks.rados.rados.0.smithi105.stdout:1007: expect (ObjNum 83 snap 1 seq_num 83) 2024-04-20T02:51:18.867 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:18.867 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:18.867+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:18.967 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:18.967+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:18.997 INFO:tasks.rados.rados.0.smithi105.stdout:1006: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.53709629999999997, 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.537119232, 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "waiting for peered", 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:19.035 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.038 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.53703085100000003, 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.53709955499999995, 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "waiting for peered", 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.039 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:19.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:51:19.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.53700535299999996, 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.53711146600000004, 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "waiting for peered", 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:19.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:51:19.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:51:19.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 3 2024-04-20T02:51:19.046 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:19.047 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:19.062 INFO:tasks.rados.rados.0.smithi105.stdout:1008: finishing set_chunk to oid 11 2024-04-20T02:51:19.062 INFO:tasks.rados.rados.0.smithi105.stdout:1008: got expected ENOENT (src dne) 2024-04-20T02:51:19.062 INFO:tasks.rados.rados.0.smithi105.stdout:996: expect (ObjNum 132 snap 9 seq_num 132) 2024-04-20T02:51:19.069 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:19.069+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:19.161 INFO:tasks.rados.rados.0.smithi105.stdout:1011: expect (ObjNum 154 snap 10 seq_num 154) 2024-04-20T02:51:19.171 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:19.171+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:19.175 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:19.175 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:19.175 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:19.175 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:19.175 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:19.186 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:19.272 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:19.273+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:19.333 INFO:tasks.rados.rados.0.smithi105.stdout:999: expect (ObjNum 151 snap 10 seq_num 151) 2024-04-20T02:51:19.357 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.3 46/44 [PushOp(4:cfd1e4d4:::smithi10548823-9:head, version: 45'100, data_included: [1817887~282988], data_size: 282988, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:cfd1e4d4:::smithi10548823-9:head@45'100, size: 2100875, copy_subset: [0~2100875], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2100875, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1817887, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-20T02:51:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:03.670012+0000", 2024-04-20T02:51:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.686616029, 2024-04-20T02:51:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0085824019999999994, 2024-04-20T02:51:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670012+0000", 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670012+0000", 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670011+0000", 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670238+0000", 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022708599999999999 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670252+0000", 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3485e-05 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670264+0000", 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2007999999999999e-05 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670313+0000", 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8925000000000003e-05 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.670322+0000", 2024-04-20T02:51:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8049999999999996e-06 2024-04-20T02:51:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:03.678594+0000", 2024-04-20T02:51:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0082727180000000001 2024-04-20T02:51:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1528 3.2 e48/47 3:7333b787:::49dfa3a36ea566ece109be00940c25f90e74796f7ebe644247ba8ecd48263ae5:head v 48'95, mlcod=0'0)", 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.934192575000001, 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088605330000000003, 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.422435+0000", 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423048+0000", 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00061281499999999995 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423060+0000", 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1644000000000001e-05 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423068+0000", 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7460000000000006e-06 2024-04-20T02:51:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423108+0000", 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.977e-05 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.423134+0000", 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5599e-05 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431261+0000", 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081278059999999996 2024-04-20T02:51:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.431296+0000", 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4413999999999999e-05 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:152 3.2 e48/47 3:54f0c00e:::2528189ec1c891f179c539fa23af2c87b9e8d2c19f70e70fac8934d3cc51ccee:head v 48'96, mlcod=0'0)", 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.923721649000001, 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020196797999999998, 2024-04-20T02:51:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433091+0000", 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000185589 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433102+0000", 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433108+0000", 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5660000000000003e-06 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439137+0000", 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060284329999999997 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439166+0000", 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9278e-05 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453075+0000", 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013909163 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453103+0000", 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7725999999999998e-05 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:154 3.2 e48/47 3:7d72d9ce:::7b4ada8765f224bb4a516820a83f51a402b4ee671c5f45bb3fbe5e6656cf1f14:head v 48'97, mlcod=42'91)", 2024-04-20T02:51:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.919816017, 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016320239, 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436811+0000", 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437600+0000", 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078939699999999995 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437613+0000", 2024-04-20T02:51:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2588e-05 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437624+0000", 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0542e-05 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442541+0000", 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049178809999999998 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442565+0000", 2024-04-20T02:51:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3558e-05 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453110+0000", 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010545172 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453132+0000", 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1929e-05 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:144 3.2 e48/47 3:7333b787:::49dfa3a36ea566ece109be00940c25f90e74796f7ebe644247ba8ecd48263ae5:head v 48'98, mlcod=42'91)", 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.438561+0000", 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.91806697, 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014604997999999999, 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438561+0000", 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438561+0000", 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438560+0000", 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438568+0000", 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3380000000000005e-06 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438583+0000", 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4284000000000001e-05 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.438591+0000", 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3569999999999998e-06 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443406+0000", 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048148610000000001 2024-04-20T02:51:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443435+0000", 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8915999999999999e-05 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453135+0000", 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0097005549999999996 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453166+0000", 2024-04-20T02:51:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0516999999999999e-05 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:155 3.2 e48/47 3:715fab8c:::b945a2927747be608263bdbe1fe814f665c73382cb3375555da5aa95a92567ff:head v 48'99, mlcod=42'91)", 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.917028962, 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017321785999999999, 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439598+0000", 2024-04-20T02:51:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439954+0000", 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00035586299999999999 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439970+0000", 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5889e-05 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439981+0000", 2024-04-20T02:51:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0754000000000001e-05 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443643+0000", 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036621259999999999 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443671+0000", 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8289e-05 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456885+0000", 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013213964999999999 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456921+0000", 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1635 3.2 e51/49 3:67667c16:::7aea211498f946c134ed12b1f5632511a2f083a0eb2114d1051f38dc7326b113:head v 51'100, mlcod=0'0)", 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 10.509138015, 2024-04-20T02:51:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019392829, 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847489+0000", 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850819+0000", 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033292069999999998 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850831+0000", 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2320000000000001e-05 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850838+0000", 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2980000000000003e-06 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850874+0000", 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5800999999999998e-05 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850890+0000", 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5536000000000002e-05 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866849+0000", 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015959346999999999 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866883+0000", 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3776999999999999e-05 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.9079897760000009, 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:51:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:51:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:51:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1664 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'77, mlcod=50'73)", 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8469107929999993, 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012476671999999999, 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509717+0000", 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509983+0000", 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00026571599999999999 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509992+0000", 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.509999+0000", 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1450000000000002e-06 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.510050+0000", 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0562000000000001e-05 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.510064+0000", 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4586000000000001e-05 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.522152+0000", 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012087736 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.522194+0000", 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1851999999999998e-05 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1665 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'78, mlcod=50'73)", 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8333899230000004, 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025918594, 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523236+0000", 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523658+0000", 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000422082 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523676+0000", 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7716000000000001e-05 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523686+0000", 2024-04-20T02:51:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9790000000000005e-06 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523734+0000", 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7579e-05 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523759+0000", 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5593e-05 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549122+0000", 2024-04-20T02:51:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025363087999999999 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549157+0000", 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4242999999999997e-05 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.9014815439999992, 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034212370999999998, 2024-04-20T02:51:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456260+0000", 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011140480000000001 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456273+0000", 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456280+0000", 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1130000000000003e-06 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456331+0000", 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1446999999999997e-05 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456357+0000", 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6001999999999999e-05 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489309+0000", 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032951972000000003 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489359+0000", 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9552000000000003e-05 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1676 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'129, mlcod=49'126)", 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.475345+0000", 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.8812828929999998, 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014576344999999999, 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.475345+0000", 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.475345+0000", 2024-04-20T02:51:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.475344+0000", 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.476165+0000", 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00082085800000000005 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.476178+0000", 2024-04-20T02:51:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3426e-05 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.476187+0000", 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4330000000000004e-06 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.476720+0000", 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053326399999999998 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.476743+0000", 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2821999999999999e-05 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489890+0000", 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013146893 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489921+0000", 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1590000000000001e-05 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1775 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'41, mlcod=53'38)", 2024-04-20T02:51:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.6031000540000004, 2024-04-20T02:51:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038672274999999999, 2024-04-20T02:51:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754868+0000", 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013404619999999999 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754881+0000", 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.323e-05 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754892+0000", 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1154e-05 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754967+0000", 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4326999999999995e-05 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.755006+0000", 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9338000000000002e-05 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792154+0000", 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037147924999999998 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792200+0000", 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6044999999999997e-05 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1776 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'42, mlcod=53'38)", 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.5831198759999996, 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019186657999999999, 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773507+0000", 2024-04-20T02:51:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774271+0000", 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076432799999999999 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774284+0000", 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2483000000000001e-05 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774294+0000", 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6569999999999992e-06 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774338+0000", 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4576000000000001e-05 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774358+0000", 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9449999999999998e-05 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792664+0000", 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018306413000000001 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792695+0000", 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0685000000000002e-05 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1822 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'94, mlcod=55'92)", 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.253077921, 2024-04-20T02:51:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031823907999999998, 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103786+0000", 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000236658 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103797+0000", 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0951000000000001e-05 2024-04-20T02:51:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103804+0000", 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0960000000000003e-06 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103860+0000", 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5781e-05 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103887+0000", 2024-04-20T02:51:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6999999999999999e-05 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135333+0000", 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031446004 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135374+0000", 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0778999999999997e-05 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1823 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'95, mlcod=55'92)", 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.2385909149999996, 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017980995999999999, 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118036+0000", 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118363+0000", 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00032734600000000001 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118381+0000", 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7816e-05 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118391+0000", 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0155e-05 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119591+0000", 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011995560000000001 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119618+0000", 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7475e-05 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135988+0000", 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016370096000000001 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.136018+0000", 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9745999999999999e-05 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1836 4.1 e58/46 4:8b927dd6:::smithi10548823-28:head v 58'99, mlcod=56'96)", 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 3.8710619130000001, 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030374581000000001, 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485930+0000", 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00036455800000000002 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485941+0000", 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1017e-05 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485950+0000", 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.85e-06 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486000+0000", 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9589999999999998e-05 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486025+0000", 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.529e-05 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515909+0000", 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029884131000000001 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515941+0000", 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1569000000000002e-05 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1837 4.1 e58/46 4:8b927dd6:::smithi10548823-28:head v 58'100, mlcod=56'96)", 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:15.501011+0000", 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 3.8556166589999998, 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015450057999999999, 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501011+0000", 2024-04-20T02:51:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501011+0000", 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501010+0000", 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501412+0000", 2024-04-20T02:51:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040223499999999997 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501426+0000", 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3801999999999999e-05 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501435+0000", 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7970000000000006e-06 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501480+0000", 2024-04-20T02:51:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5695999999999998e-05 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501505+0000", 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4213999999999999e-05 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.516435+0000", 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014930047 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.516461+0000", 2024-04-20T02:51:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6571999999999999e-05 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1891 4.17 e61/60 4:e9876a63:::smithi10548823-30:head v 61'45, mlcod=0'0)", 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.843170266, 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018777245000000001, 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513457+0000", 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514596+0000", 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011393289999999999 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514612+0000", 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6311e-05 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514623+0000", 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056999999999999e-05 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514670+0000", 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6557999999999997e-05 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514698+0000", 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8121000000000001e-05 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.532204+0000", 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017505969999999999 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.532235+0000", 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0731000000000001e-05 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1892 4.17 e61/60 4:e9876a63:::smithi10548823-30:head v 61'46, mlcod=0'0)", 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.81983128199999999, 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020303921999999999, 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:19.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536796+0000", 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537578+0000", 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078217499999999997 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537592+0000", 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3721e-05 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537602+0000", 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0380000000000001e-05 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537638+0000", 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5559000000000003e-05 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537657+0000", 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9242e-05 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.557064+0000", 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019406970999999999 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:19.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:19.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:19.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.557101+0000", 2024-04-20T02:51:19.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6572000000000002e-05 2024-04-20T02:51:19.442 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.442 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.442 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.442 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:19.442 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:19.442 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:19.444 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:19.445 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:19.443+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:19.491 INFO:tasks.rados.rados.0.smithi105.stdout:995: expect (ObjNum 133 snap 9 seq_num 133) 2024-04-20T02:51:19.508 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:51:19.544 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:19.544+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:19.585 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:19.585 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:19.585 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:19.585 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:19.595 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:19.626 INFO:tasks.rados.rados.0.smithi105.stdout:1012: finishing rollback tid 0 to smithi10548823-41 2024-04-20T02:51:19.626 INFO:tasks.rados.rados.0.smithi105.stdout:1012: finishing rollback tid 1 to smithi10548823-41 2024-04-20T02:51:19.626 INFO:tasks.rados.rados.0.smithi105.stdout:995: done (15 left) 2024-04-20T02:51:19.626 INFO:tasks.rados.rados.0.smithi105.stdout:996: done (14 left) 2024-04-20T02:51:19.626 INFO:tasks.rados.rados.0.smithi105.stdout:997: done (13 left) 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:998: done (12 left) 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:999: done (11 left) 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:1006: done (10 left) 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:1007: done (9 left) 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:1008: done (8 left) 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:1010: done (7 left) 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:1011: done (6 left) 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:1013: write oid 22 current snap is 14 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:1012: finishing rollback tid 2 to smithi10548823-41 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 157 (ObjNum 87 snap 2 seq_num 87) dirty exists 2024-04-20T02:51:19.627 INFO:tasks.rados.rados.0.smithi105.stdout:1002: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:51:19.645 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:19.646+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:19.746 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:19.746+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:19.756 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:19.756 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:19.756 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:19.756 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:19.757 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:19.767 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:19.807 INFO:tasks.rados.rados.0.smithi105.stdout:1003: finishing write tid 1 to smithi10548823-40 2024-04-20T02:51:19.807 INFO:tasks.rados.rados.0.smithi105.stdout:1003: finishing write tid 2 to smithi10548823-40 2024-04-20T02:51:19.807 INFO:tasks.rados.rados.0.smithi105.stdout:1013: seq_num 163 ranges {739541=709000,2121859=426376} 2024-04-20T02:51:19.815 INFO:tasks.rados.rados.0.smithi105.stdout:1013: writing smithi10548823-22 from 739541 to 1448541 tid 1 2024-04-20T02:51:19.822 INFO:tasks.rados.rados.0.smithi105.stdout:1013: writing smithi10548823-22 from 2121859 to 2548235 tid 2 2024-04-20T02:51:19.822 INFO:tasks.rados.rados.0.smithi105.stdout:1002: done (6 left) 2024-04-20T02:51:19.822 INFO:tasks.rados.rados.0.smithi105.stdout:1012: done (5 left) 2024-04-20T02:51:19.823 INFO:tasks.rados.rados.0.smithi105.stdout:1014: delete oid 26 current snap is 14 2024-04-20T02:51:19.823 INFO:tasks.rados.rados.0.smithi105.stdout:1003: finishing write tid 3 to smithi10548823-40 2024-04-20T02:51:19.823 INFO:tasks.rados.rados.0.smithi105.stdout:1003: finishing write tid 4 to smithi10548823-40 2024-04-20T02:51:19.823 INFO:tasks.rados.rados.0.smithi105.stdout:1003: oid 40 updating version 0 to 155 2024-04-20T02:51:19.823 INFO:tasks.rados.rados.0.smithi105.stdout:1003: oid 40 version 155 is already newer than 154 2024-04-20T02:51:19.823 INFO:tasks.rados.rados.0.smithi105.stdout:1003: oid 40 updating version 155 to 156 2024-04-20T02:51:19.823 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 156 (ObjNum 161 snap 13 seq_num 161) dirty exists 2024-04-20T02:51:19.824 INFO:tasks.rados.rados.0.smithi105.stdout:1003: left oid 40 (ObjNum 161 snap 13 seq_num 161) 2024-04-20T02:51:19.824 INFO:tasks.rados.rados.0.smithi105.stdout:1004: finishing write tid 1 to smithi10548823-30 2024-04-20T02:51:19.824 INFO:tasks.rados.rados.0.smithi105.stdout:1004: finishing write tid 2 to smithi10548823-30 2024-04-20T02:51:19.824 INFO:tasks.rados.rados.0.smithi105.stdout:1004: finishing write tid 3 to smithi10548823-30 2024-04-20T02:51:19.824 INFO:tasks.rados.rados.0.smithi105.stdout:1004: finishing write tid 4 to smithi10548823-30 2024-04-20T02:51:19.824 INFO:tasks.rados.rados.0.smithi105.stdout:1004: oid 30 updating version 0 to 45 2024-04-20T02:51:19.824 INFO:tasks.rados.rados.0.smithi105.stdout:1004: oid 30 updating version 45 to 47 2024-04-20T02:51:19.824 INFO:tasks.rados.rados.0.smithi105.stdout:1004: oid 30 version 47 is already newer than 46 2024-04-20T02:51:19.824 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 47 (ObjNum 162 snap 13 seq_num 162) dirty exists 2024-04-20T02:51:19.824 INFO:tasks.rados.rados.0.smithi105.stdout:1004: left oid 30 (ObjNum 162 snap 13 seq_num 162) 2024-04-20T02:51:19.847 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:19.848+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:19.897 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:927 4.3 4:cf11ccc0:::smithi10548823-4:head [write 3121996~104915] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.518678887, 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058986447999999997, 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.898 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.899 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.899 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 927 2024-04-20T02:51:19.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.899 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:19.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:19.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377548+0000", 2024-04-20T02:51:19.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:19.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377607+0000", 2024-04-20T02:51:19.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9413999999999997e-05 2024-04-20T02:51:19.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377616+0000", 2024-04-20T02:51:19.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4880000000000007e-06 2024-04-20T02:51:19.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377621+0000", 2024-04-20T02:51:19.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.682e-06 2024-04-20T02:51:19.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407778+0000", 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030156110999999999 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407866+0000", 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8645000000000003e-05 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407985+0000", 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118874 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411741+0000", 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037563060000000001 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411854+0000", 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112898 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436438+0000", 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024583885 2024-04-20T02:51:19.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436483+0000", 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5334999999999999e-05 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436535+0000", 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1780000000000002e-05 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.906 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:928 4.3 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (55) in=7b,truncate 3226911] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.518600280999998, 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059064512999999999, 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 928 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377630+0000", 2024-04-20T02:51:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7259999999999998e-06 2024-04-20T02:51:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377633+0000", 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1489999999999998e-06 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377636+0000", 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7760000000000002e-06 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410754+0000", 2024-04-20T02:51:19.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033117944000000003 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410850+0000", 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6522999999999997e-05 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411008+0000", 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158157 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436151+0000", 2024-04-20T02:51:19.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025142813 2024-04-20T02:51:19.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:19.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436160+0000", 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1460000000000002e-06 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436588+0000", 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00042776499999999998 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436631+0000", 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3084000000000003e-05 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436692+0000", 2024-04-20T02:51:19.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0829000000000003e-05 2024-04-20T02:51:19.914 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.914 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.914 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.914 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:929 4.3 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.518589140000003, 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059214058, 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 929 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377638+0000", 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377640+0000", 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5430000000000001e-06 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377643+0000", 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0900000000000001e-06 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377647+0000", 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2230000000000001e-06 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411271+0000", 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033623635999999998 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411329+0000", 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8043e-05 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436728+0000", 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025399300999999999 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436793+0000", 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5041000000000003e-05 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436853+0000", 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9446e-05 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:936 4.3 4:ca036f62:::smithi10548823-34:head [write 443718~615603] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:19.919 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.501553807000001, 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060880228000000002, 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 936 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:19.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:19.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394672+0000", 2024-04-20T02:51:19.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:19.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394825+0000", 2024-04-20T02:51:19.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015343299999999999 2024-04-20T02:51:19.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394844+0000", 2024-04-20T02:51:19.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8414999999999999e-05 2024-04-20T02:51:19.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394859+0000", 2024-04-20T02:51:19.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4738e-05 2024-04-20T02:51:19.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411879+0000", 2024-04-20T02:51:19.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017020469999999999 2024-04-20T02:51:19.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411954+0000", 2024-04-20T02:51:19.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5239000000000002e-05 2024-04-20T02:51:19.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:19.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412104+0000", 2024-04-20T02:51:19.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014944399999999999 2024-04-20T02:51:19.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436402+0000", 2024-04-20T02:51:19.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024297828 2024-04-20T02:51:19.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:19.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436409+0000", 2024-04-20T02:51:19.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1779999999999997e-06 2024-04-20T02:51:19.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455390+0000", 2024-04-20T02:51:19.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018981221 2024-04-20T02:51:19.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455508+0000", 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117556 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455554+0000", 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6570000000000003e-05 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.929 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:937 4.3 4:ca036f62:::smithi10548823-34:head [write 1564138~470341] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.496215477, 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060119655000000001, 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 937 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400010+0000", 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:19.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400222+0000", 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000211208 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400238+0000", 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419e-05 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400250+0000", 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2091e-05 2024-04-20T02:51:19.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425851+0000", 2024-04-20T02:51:19.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025600795999999999 2024-04-20T02:51:19.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425965+0000", 2024-04-20T02:51:19.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011411799999999999 2024-04-20T02:51:19.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426134+0000", 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168517 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436937+0000", 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01080328 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:19.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455351+0000", 2024-04-20T02:51:19.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018414607 2024-04-20T02:51:19.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460001+0000", 2024-04-20T02:51:19.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046491809999999996 2024-04-20T02:51:19.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460086+0000", 2024-04-20T02:51:19.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5066000000000002e-05 2024-04-20T02:51:19.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460132+0000", 2024-04-20T02:51:19.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6174999999999998e-05 2024-04-20T02:51:19.937 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.937 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.937 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.937 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:938 4.3 4:ca036f62:::smithi10548823-34:head [write 2633066~610652] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:19.937 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:19.938 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.489450021000003, 2024-04-20T02:51:19.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054360654000000001, 2024-04-20T02:51:19.938 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.938 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.938 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.938 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.938 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 938 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406775+0000", 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406947+0000", 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171619 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406962+0000", 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5248999999999999e-05 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406979+0000", 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7334999999999999e-05 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436954+0000", 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029974218 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437007+0000", 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3553e-05 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437110+0000", 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455605+0000", 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018494295000000001 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455829+0000", 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022443100000000001 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461044+0000", 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052150499999999997 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461090+0000", 2024-04-20T02:51:19.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6100999999999998e-05 2024-04-20T02:51:19.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461138+0000", 2024-04-20T02:51:19.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8003000000000001e-05 2024-04-20T02:51:19.944 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:939 4.3 4:ca036f62:::smithi10548823-34:head [setxattr _header (56) in=7b,truncate 3243718] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.489237709999998, 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054380381999999998, 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.945 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 939 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406989+0000", 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406993+0000", 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9400000000000004e-06 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406999+0000", 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4389999999999999e-06 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407008+0000", 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3190000000000001e-06 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.454982+0000", 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047973729999999999 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455051+0000", 2024-04-20T02:51:19.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9197999999999997e-05 2024-04-20T02:51:19.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:19.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455157+0000", 2024-04-20T02:51:19.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106302 2024-04-20T02:51:19.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456120+0000", 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096236599999999998 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456272+0000", 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000152573 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461274+0000", 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005002055 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461318+0000", 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3282999999999999e-05 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461370+0000", 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2704999999999998e-05 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.951 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:940 4.3 4:ca036f62:::smithi10548823-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.489213161000002, 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054543401999999998, 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 940 2024-04-20T02:51:19.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407014+0000", 2024-04-20T02:51:19.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407017+0000", 2024-04-20T02:51:19.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9790000000000002e-06 2024-04-20T02:51:19.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407022+0000", 2024-04-20T02:51:19.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5539999999999997e-06 2024-04-20T02:51:19.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407027+0000", 2024-04-20T02:51:19.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6099999999999999e-06 2024-04-20T02:51:19.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455643+0000", 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.048616331999999998 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455714+0000", 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0610000000000003e-05 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461403+0000", 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056888219999999996 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461480+0000", 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7342000000000006e-05 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461558+0000", 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7780000000000004e-05 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.957 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.351567625999998, 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:19.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:19.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:51:19.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:51:19.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:51:19.962 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.963645164999999, 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.963 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.964 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:51:19.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.964 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:19.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:19.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:51:19.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:19.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:51:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:51:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:51:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:51:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:51:19.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:51:19.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:19.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:51:19.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:51:19.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:51:19.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:51:19.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:51:19.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:51:19.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.963384958999999, 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:51:19.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:51:19.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:51:19.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:51:19.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:51:19.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:51:19.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:51:19.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:19.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:51:19.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:51:19.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.963363490999999, 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.978 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:51:19.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:51:19.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:51:19.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:51:19.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:51:19.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:51:19.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:19.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:51:19.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:51:19.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:51:19.984 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.769691559000002, 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:19.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.990 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:133 3.1 3:8efdf53d:::0188443c5b8894a8247535d52a547bece3a4c5f490fb220852b80e77d71ddcda:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.768967036999999, 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060974049000000002, 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:19.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127260+0000", 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127435+0000", 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017459800000000001 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127443+0000", 2024-04-20T02:51:19.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3790000000000006e-06 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127450+0000", 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5089999999999999e-06 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.129973+0000", 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025229639999999999 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130090+0000", 2024-04-20T02:51:19.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117521 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130323+0000", 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023271199999999999 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.147581+0000", 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017258440999999999 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:19.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155339+0000", 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077577189999999997 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188139+0000", 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032800425000000001 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188195+0000", 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5773999999999998e-05 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188235+0000", 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9427e-05 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:136 3.1 3:a965252c:::a490375a0954219328244dfa32cf9967dfac8ed04667cb9b5ce15f4910db2ee6:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.768771820000001, 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060878320999999999, 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:19.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127455+0000", 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:19.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128197+0000", 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074176999999999999 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128208+0000", 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0498e-05 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128216+0000", 2024-04-20T02:51:19.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2630000000000003e-06 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135797+0000", 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0075808669999999998 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135930+0000", 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013299000000000001 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.136192+0000", 2024-04-20T02:51:19.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026213899999999999 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148053+0000", 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011860723 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155346+0000", 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0072930039999999996 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188270+0000", 2024-04-20T02:51:20.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032924299999999997 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188298+0000", 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8041000000000001e-05 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188334+0000", 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6247e-05 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:139 3.1 3:90772484:::f9b8a83c02fab1c87f6aef9574ab68b1d6a75b657e100bac3693f85cac340cb2:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.762113894999999, 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054361675999999998, 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:20.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134113+0000", 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:20.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134645+0000", 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053281300000000004 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134662+0000", 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6503000000000001e-05 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134672+0000", 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8090000000000004e-06 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148073+0000", 2024-04-20T02:51:20.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013401091 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148177+0000", 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104344 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148417+0000", 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023968899999999999 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155385+0000", 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0069679260000000002 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155725+0000", 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034046800000000002 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188383+0000", 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032658154000000002 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:20.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188433+0000", 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0059999999999998e-05 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188475+0000", 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2049999999999999e-05 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:140 3.1 3:b80bf598:::82930be3d6fd0d013eae1f4ab54a437986489fd06a4f4934c2432c14687252ab:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.761548640000001, 2024-04-20T02:51:20.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053882434, 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:20.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134678+0000", 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134756+0000", 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7527999999999994e-05 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134763+0000", 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.092e-06 2024-04-20T02:51:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0290000000000002e-06 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152310+0000", 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017540581999999999 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152431+0000", 2024-04-20T02:51:20.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012166 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152621+0000", 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000189944 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155992+0000", 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033703470000000001 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:20.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160642+0000", 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046505239999999996 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188508+0000", 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027866134000000001 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188534+0000", 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5945999999999999e-05 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188561+0000", 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7300999999999998e-05 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1799 4.6 4:697da67d:::smithi10548823-48:head [write 588278~715406,stat] snapc 13=[13,d] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.8226234850000003, 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053216977999999998, 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:20.013 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1799 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:20.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073603+0000", 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073742+0000", 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000138727 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:20.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073754+0000", 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1970000000000001e-05 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073761+0000", 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.883e-06 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073806+0000", 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5840999999999997e-05 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073872+0000", 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5520999999999998e-05 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074005+0000", 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013345499999999999 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124313+0000", 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050307628 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124361+0000", 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7608999999999998e-05 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126705+0000", 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0023443790000000002 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126770+0000", 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4737e-05 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126821+0000", 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1375000000000002e-05 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1801 4.6 4:697da67d:::smithi10548823-48:head [setxattr _header (57) in=7b,truncate 2588278] snapc 13=[13,d] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.8164485319999999, 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053362723000000001, 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:20.019 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1801 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079782+0000", 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7840000000000001e-06 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079790+0000", 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3270000000000003e-06 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079799+0000", 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5040000000000008e-06 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123583+0000", 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043783913000000001 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123714+0000", 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013130699999999999 2024-04-20T02:51:20.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123889+0000", 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017436200000000001 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132488+0000", 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085989889999999996 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:20.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132702+0000", 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021448600000000001 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133026+0000", 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032330500000000002 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133079+0000", 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3127000000000001e-05 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:20.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133142+0000", 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3150000000000004e-05 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1802 4.6 4:697da67d:::smithi10548823-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.8164148469999999, 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053694407999999999, 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:20.025 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1802 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:20.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079812+0000", 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079815+0000", 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1329999999999999e-06 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079823+0000", 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.401e-06 2024-04-20T02:51:20.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079828+0000", 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4149999999999998e-06 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124387+0000", 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044558788000000002 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124468+0000", 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1447999999999997e-05 2024-04-20T02:51:20.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133186+0000", 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0087175610000000004 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133344+0000", 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158393 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133507+0000", 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000162829 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.029 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:20.030 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.030 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:20.030 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:20.030 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:20.030 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:20.032 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:20.032 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:20.031+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:20.132 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:20.133+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:20.205 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:20.205 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:20.205 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:20.205 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:20.216 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:20.219 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:20.233 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:20.234+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:20.242 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:20.262 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:20.281 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:20.301 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:20.320 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:20.335 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:20.335+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:20.339 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:20.347 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:20.348 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:20.359 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:20.360 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:20.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:20.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:20.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:20.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:20.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:20.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:20.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:20.363 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:20.435 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:20.436+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:20.489 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:20.489 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:20.498 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:51:20.498 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:51:20.498 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-20T02:51:20.498 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T02:51:20.498 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T02:51:20.498 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-04-20T02:51:20.504 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:20.543 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:20.544+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:20.629 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:20.629 INFO:teuthology.orchestra.run.smithi138.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T02:51:20.629 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:20.631 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:20.631 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:20.637 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:20.637+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:20.642 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:20.737 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:20.738+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:20.809 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:20.809 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:20.819 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:20.863 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:20.863+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:20.940 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:20.941+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:20.946 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:20.946 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:20.956 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:21.042 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:21.042+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:21.124 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:21.125 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:21.135 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:21.180 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:21.180+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:21.243 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:21.244+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:21.263 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:21.263 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:21.273 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:21.344 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:21.344+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:21.444 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:21.444 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:21.444 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:21.444+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:21.454 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:21.507 INFO:tasks.rados.rados.0.smithi105.stdout:1003: done (5 left) 2024-04-20T02:51:21.507 INFO:tasks.rados.rados.0.smithi105.stdout:1004: done (4 left) 2024-04-20T02:51:21.507 INFO:tasks.rados.rados.0.smithi105.stdout:1014: done (3 left) 2024-04-20T02:51:21.507 INFO:tasks.rados.rados.0.smithi105.stdout:1015: read oid 43 snap -1 2024-04-20T02:51:21.507 INFO:tasks.rados.rados.0.smithi105.stdout:1015: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:51:21.507 INFO:tasks.rados.rados.0.smithi105.stdout:1016: read oid 14 snap 11 2024-04-20T02:51:21.507 INFO:tasks.rados.rados.0.smithi105.stdout:1016: expect (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:51:21.507 INFO:tasks.rados.rados.0.smithi105.stdout:1017: write oid 20 current snap is 14 2024-04-20T02:51:21.507 INFO:tasks.rados.rados.0.smithi105.stdout:1017: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-04-20T02:51:21.516 INFO:tasks.rados.rados.0.smithi105.stdout:1017: writing smithi10548823-20 from 773155 to 1232653 tid 1 2024-04-20T02:51:21.521 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:51:21.534 INFO:tasks.rados.rados.0.smithi105.stdout:1017: writing smithi10548823-20 from 1722470 to 2152515 tid 2 2024-04-20T02:51:21.535 INFO:tasks.rados.rados.0.smithi105.stdout:1017: writing smithi10548823-20 from 2648676 to 2773155 tid 3 2024-04-20T02:51:21.535 INFO:tasks.rados.rados.0.smithi105.stdout:1015: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-20T02:51:21.545 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:21.545+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:21.620 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:21.620 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:21.632 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:21.645 INFO:tasks.rados.rados.0.smithi105.stdout:1015: done (5 left) 2024-04-20T02:51:21.645 INFO:tasks.rados.rados.0.smithi105.stdout:1018: write oid 43 current snap is 14 2024-04-20T02:51:21.645 INFO:tasks.rados.rados.0.smithi105.stdout:1018: seq_num 166 ranges {789962=534747,1780952=690056} 2024-04-20T02:51:21.653 INFO:tasks.rados.rados.0.smithi105.stdout:1018: writing smithi10548823-43 from 789962 to 1324709 tid 1 2024-04-20T02:51:21.659 INFO:tasks.rados.rados.0.smithi105.stdout:1018: writing smithi10548823-43 from 1780952 to 2471008 tid 2 2024-04-20T02:51:21.659 INFO:tasks.rados.rados.0.smithi105.stdout:1019: read oid 30 snap 11 2024-04-20T02:51:21.659 INFO:tasks.rados.rados.0.smithi105.stdout:1019: expect (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:51:21.660 INFO:tasks.rados.rados.0.smithi105.stdout:1020: write oid 12 current snap is 14 2024-04-20T02:51:21.660 INFO:tasks.rados.rados.0.smithi105.stdout:1020: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-04-20T02:51:21.668 INFO:tasks.rados.rados.0.smithi105.stdout:1020: writing smithi10548823-12 from 406769 to 1016765 tid 1 2024-04-20T02:51:21.673 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:21.674+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:21.675 INFO:tasks.rados.rados.0.smithi105.stdout:1020: writing smithi10548823-12 from 1439434 to 1873148 tid 2 2024-04-20T02:51:21.676 INFO:tasks.rados.rados.0.smithi105.stdout:1020: writing smithi10548823-12 from 2303522 to 2806769 tid 3 2024-04-20T02:51:21.685 INFO:tasks.rados.rados.0.smithi105.stdout:1021: write (excl) oid 11 current snap is 14 2024-04-20T02:51:21.686 INFO:tasks.rados.rados.0.smithi105.stdout:1021: seq_num 168 ranges {423576=401598,1497916=693725} 2024-04-20T02:51:21.686 INFO:tasks.rados.rados.0.smithi105.stdout:1021: writing smithi10548823-11 from 423576 to 825174 tid 1 2024-04-20T02:51:21.697 INFO:tasks.rados.rados.0.smithi105.stdout:1021: writing smithi10548823-11 from 1497916 to 2191641 tid 2 2024-04-20T02:51:21.697 INFO:tasks.rados.rados.0.smithi105.stdout:1022: write (excl) oid 25 current snap is 14 2024-04-20T02:51:21.697 INFO:tasks.rados.rados.0.smithi105.stdout:1022: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-04-20T02:51:21.703 INFO:tasks.rados.rados.0.smithi105.stdout:1022: writing smithi10548823-25 from 440383 to 917230 tid 1 2024-04-20T02:51:21.704 INFO:tasks.rados.rados.0.smithi105.stdout:1022: writing smithi10548823-25 from 1556398 to 1993781 tid 2 2024-04-20T02:51:21.711 INFO:tasks.rados.rados.0.smithi105.stdout:1022: writing smithi10548823-25 from 2474721 to 2840383 tid 3 2024-04-20T02:51:21.711 INFO:tasks.rados.rados.0.smithi105.stdout:1023: write oid 34 current snap is 14 2024-04-20T02:51:21.711 INFO:tasks.rados.rados.0.smithi105.stdout:1023: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-04-20T02:51:21.715 INFO:tasks.rados.rados.0.smithi105.stdout:1023: writing smithi10548823-34 from 457190 to 1009286 tid 1 2024-04-20T02:51:21.722 INFO:tasks.rados.rados.0.smithi105.stdout:1023: writing smithi10548823-34 from 1614880 to 2195921 tid 2 2024-04-20T02:51:21.726 INFO:tasks.rados.rados.0.smithi105.stdout:1023: writing smithi10548823-34 from 2785791 to 2857190 tid 3 2024-04-20T02:51:21.726 INFO:tasks.rados.rados.0.smithi105.stdout:1024: write (excl) oid 44 current snap is 14 2024-04-20T02:51:21.726 INFO:tasks.rados.rados.0.smithi105.stdout:1024: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-04-20T02:51:21.733 INFO:tasks.rados.rados.0.smithi105.stdout:1024: writing smithi10548823-44 from 473997 to 1101342 tid 1 2024-04-20T02:51:21.739 INFO:tasks.rados.rados.0.smithi105.stdout:1024: writing smithi10548823-44 from 1673362 to 2114414 tid 2 2024-04-20T02:51:21.756 INFO:tasks.rados.rados.0.smithi105.stdout:1024: writing smithi10548823-44 from 2529567 to 2873997 tid 3 2024-04-20T02:51:21.756 INFO:tasks.rados.rados.0.smithi105.stdout:1025: write (excl) oid 45 current snap is 14 2024-04-20T02:51:21.756 INFO:tasks.rados.rados.0.smithi105.stdout:1025: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-04-20T02:51:21.765 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:21.756+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:21.766 INFO:tasks.rados.rados.0.smithi105.stdout:1025: writing smithi10548823-45 from 490804 to 1193398 tid 1 2024-04-20T02:51:21.767 INFO:tasks.rados.rados.0.smithi105.stdout:1025: writing smithi10548823-45 from 1615491 to 2200201 tid 2 2024-04-20T02:51:21.771 INFO:tasks.rados.rados.0.smithi105.stdout:1025: writing smithi10548823-45 from 2724284 to 2890804 tid 3 2024-04-20T02:51:21.771 INFO:tasks.rados.rados.0.smithi105.stdout:1026: read oid 49 snap -1 2024-04-20T02:51:21.771 INFO:tasks.rados.rados.0.smithi105.stdout:1026: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:21.771 INFO:tasks.rados.rados.0.smithi105.stdout:1027: tier_evict oid 40 2024-04-20T02:51:21.771 INFO:tasks.rados.rados.0.smithi105.stdout:1027: tier_evict oid 40 snap -1 2024-04-20T02:51:21.772 INFO:tasks.rados.rados.0.smithi105.stdout:1028: read oid 23 snap 13 2024-04-20T02:51:21.772 INFO:tasks.rados.rados.0.smithi105.stdout:1028: expect (ObjNum 130 snap 9 seq_num 130) 2024-04-20T02:51:21.772 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:21.781 INFO:tasks.rados.rados.0.smithi105.stdout:1016: expect (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:51:21.816 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:21.816 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:21.830 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:21.874 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:21.875+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:21.948 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:21.948+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:21.981 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:21.981 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:21.992 INFO:tasks.rados.rados.0.smithi105.stdout:1009: expect (ObjNum 86 snap 2 seq_num 86) 2024-04-20T02:51:21.996 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:22.081 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:22.082+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:22.145 INFO:tasks.rados.rados.0.smithi105.stdout:1001: got (0) Success 2024-04-20T02:51:22.145 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 0 (ObjNum 123 snap 7 seq_num 123) dirty exists 2024-04-20T02:51:22.145 INFO:tasks.rados.rados.0.smithi105.stdout:1001: done (15 left) 2024-04-20T02:51:22.145 INFO:tasks.rados.rados.0.smithi105.stdout:1009: done (14 left) 2024-04-20T02:51:22.145 INFO:tasks.rados.rados.0.smithi105.stdout:1016: done (13 left) 2024-04-20T02:51:22.146 INFO:tasks.rados.rados.0.smithi105.stdout:1029: write oid 18 current snap is 14 2024-04-20T02:51:22.146 INFO:tasks.rados.rados.0.smithi105.stdout:1013: finishing write tid 1 to smithi10548823-22 2024-04-20T02:51:22.146 INFO:tasks.rados.rados.0.smithi105.stdout:1013: finishing write tid 2 to smithi10548823-22 2024-04-20T02:51:22.146 INFO:tasks.rados.rados.0.smithi105.stdout:1013: finishing write tid 3 to smithi10548823-22 2024-04-20T02:51:22.146 INFO:tasks.rados.rados.0.smithi105.stdout:1029: seq_num 173 ranges {507611=777843,2073973=444721} 2024-04-20T02:51:22.150 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:22.151+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:22.151 INFO:tasks.rados.rados.0.smithi105.stdout:1029: writing smithi10548823-18 from 507611 to 1285454 tid 1 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:1029: writing smithi10548823-18 from 2073973 to 2518694 tid 2 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:1030: tier_promote oid 33 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:1013: finishing write tid 4 to smithi10548823-22 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:1013: oid 22 updating version 0 to 61 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:1013: oid 22 version 61 is already newer than 59 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:1013: oid 22 version 61 is already newer than 60 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 61 (ObjNum 163 snap 14 seq_num 163) dirty exists 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:1013: left oid 22 (ObjNum 163 snap 14 seq_num 163) 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:1017: finishing write tid 1 to smithi10548823-20 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:1017: finishing write tid 2 to smithi10548823-20 2024-04-20T02:51:22.160 INFO:tasks.rados.rados.0.smithi105.stdout:1017: finishing write tid 3 to smithi10548823-20 2024-04-20T02:51:22.161 INFO:tasks.rados.rados.0.smithi105.stdout:1017: finishing write tid 4 to smithi10548823-20 2024-04-20T02:51:22.161 INFO:tasks.rados.rados.0.smithi105.stdout:1013: done (14 left) 2024-04-20T02:51:22.161 INFO:tasks.rados.rados.0.smithi105.stdout:1031: write (excl) oid 32 current snap is 14 2024-04-20T02:51:22.161 INFO:tasks.rados.rados.0.smithi105.stdout:1031: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-04-20T02:51:22.163 INFO:tasks.rados.rados.0.smithi105.stdout:1031: writing smithi10548823-32 from 524418 to 977510 tid 1 2024-04-20T02:51:22.171 INFO:tasks.rados.rados.0.smithi105.stdout:1031: writing smithi10548823-32 from 1732455 to 2320834 tid 2 2024-04-20T02:51:22.175 INFO:tasks.rados.rados.0.smithi105.stdout:1031: writing smithi10548823-32 from 2779130 to 2924418 tid 3 2024-04-20T02:51:22.175 INFO:tasks.rados.rados.0.smithi105.stdout:1032: write oid 7 current snap is 14 2024-04-20T02:51:22.175 INFO:tasks.rados.rados.0.smithi105.stdout:1017: finishing write tid 5 to smithi10548823-20 2024-04-20T02:51:22.175 INFO:tasks.rados.rados.0.smithi105.stdout:1017: oid 20 updating version 0 to 77 2024-04-20T02:51:22.175 INFO:tasks.rados.rados.0.smithi105.stdout:1017: oid 20 updating version 77 to 79 2024-04-20T02:51:22.175 INFO:tasks.rados.rados.0.smithi105.stdout:1017: oid 20 updating version 79 to 80 2024-04-20T02:51:22.175 INFO:tasks.rados.rados.0.smithi105.stdout:1017: oid 20 version 80 is already newer than 78 2024-04-20T02:51:22.175 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 80 (ObjNum 165 snap 14 seq_num 165) dirty exists 2024-04-20T02:51:22.175 INFO:tasks.rados.rados.0.smithi105.stdout:1017: left oid 20 (ObjNum 165 snap 14 seq_num 165) 2024-04-20T02:51:22.176 INFO:tasks.rados.rados.0.smithi105.stdout:1032: seq_num 175 ranges {541225=644694,1907290=448390} 2024-04-20T02:51:22.177 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:22.177 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:22.181 INFO:tasks.rados.rados.0.smithi105.stdout:1032: writing smithi10548823-7 from 541225 to 1185919 tid 1 2024-04-20T02:51:22.188 INFO:tasks.rados.rados.0.smithi105.stdout:1032: writing smithi10548823-7 from 1907290 to 2355680 tid 2 2024-04-20T02:51:22.188 INFO:tasks.rados.rados.0.smithi105.stdout:1017: done (15 left) 2024-04-20T02:51:22.188 INFO:tasks.rados.rados.0.smithi105.stdout:1033: set_chunk oid 31 target oid 2024-04-20T02:51:22.194 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:22.261 INFO:tasks.rados.rados.0.smithi105.stdout:1033: set_chunk oid 31 offset: 578048 length: 1196032 target oid 0aa9fbb5553cebb4b59d0ead1366c9df0a4e80a5b7ff81a1dc44fb5e13a51153 offset: 0 snap -1 2024-04-20T02:51:22.261 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:22.261 INFO:tasks.rados.rados.0.smithi105.stdout:1019: expect (ObjNum 107 snap 5 seq_num 107) 2024-04-20T02:51:22.278 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:22.279+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:22.352 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:22.352+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:22.363 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:22.363 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:22.373 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:22.375 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:22.453 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:22.453+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:22.522 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:22.540 INFO:tasks.rados.rados.0.smithi105.stdout:1027: got (22) Invalid argument 2024-04-20T02:51:22.541 INFO:tasks.rados.rados.0.smithi105.stdout:1018: finishing write tid 1 to smithi10548823-43 2024-04-20T02:51:22.541 INFO:tasks.rados.rados.0.smithi105.stdout:1022: finishing write tid 1 to smithi10548823-25 2024-04-20T02:51:22.541 INFO:tasks.rados.rados.0.smithi105.stdout:1019: done (15 left) 2024-04-20T02:51:22.541 INFO:tasks.rados.rados.0.smithi105.stdout:1027: done (14 left) 2024-04-20T02:51:22.541 INFO:tasks.rados.rados.0.smithi105.stdout:1034: read oid 5 snap -1 2024-04-20T02:51:22.541 INFO:tasks.rados.rados.0.smithi105.stdout:1034: expect (ObjNum 144 snap 9 seq_num 144) 2024-04-20T02:51:22.541 INFO:tasks.rados.rados.0.smithi105.stdout:1035: read oid 15 snap -1 2024-04-20T02:51:22.541 INFO:tasks.rados.rados.0.smithi105.stdout:1035: expect deleted 2024-04-20T02:51:22.541 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:22.541 INFO:tasks.rados.rados.0.smithi105.stdout:1028: expect (ObjNum 130 snap 9 seq_num 130) 2024-04-20T02:51:22.542 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:22.542 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:22.543 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:22.554 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:22.554+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:22.557 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:22.563 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:51:22.569 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:22.589 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:22.608 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:22.628 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:22.647 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:22.655 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:22.656+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:22.663 INFO:tasks.rados.rados.0.smithi105.stdout:1026: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:22.666 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:22.667 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:22.667 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:22.667 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:22.668 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:22.668 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:22.668 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:22.668 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:22.668 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:22.688 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:22.688 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:22.688 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:22.688 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:22.698 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:22.782 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:22.783+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:22.804 INFO:tasks.rados.rados.0.smithi105.stdout:1018: finishing write tid 2 to smithi10548823-43 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1018: finishing write tid 3 to smithi10548823-43 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1020: finishing write tid 1 to smithi10548823-12 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1018: finishing write tid 4 to smithi10548823-43 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1018: oid 43 updating version 0 to 70 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1018: oid 43 updating version 70 to 71 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1018: oid 43 version 71 is already newer than 69 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 71 (ObjNum 166 snap 14 seq_num 166) dirty exists 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1018: left oid 43 (ObjNum 166 snap 14 seq_num 166) 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1021: finishing write tid 1 to smithi10548823-11 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1018: done (15 left) 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1026: done (14 left) 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1028: done (13 left) 2024-04-20T02:51:22.805 INFO:tasks.rados.rados.0.smithi105.stdout:1036: set_chunk oid 40 target oid 2024-04-20T02:51:22.824 INFO:tasks.rados.rados.0.smithi105.stdout:1036: set_chunk oid 40 offset: 1863680 length: 736256 target oid 697e00678e8294241cc2f73256a258988cd5fe8174d6ddfa5b6c8b4cff1d019a offset: 0 snap -1 2024-04-20T02:51:22.825 INFO:tasks.rados.rados.0.smithi105.stdout:1037: write (excl) oid 43 current snap is 14 2024-04-20T02:51:22.825 INFO:tasks.rados.rados.0.smithi105.stdout:1020: finishing write tid 2 to smithi10548823-12 2024-04-20T02:51:22.825 INFO:tasks.rados.rados.0.smithi105.stdout:1021: finishing write tid 2 to smithi10548823-11 2024-04-20T02:51:22.825 INFO:tasks.rados.rados.0.smithi105.stdout:1021: finishing write tid 3 to smithi10548823-11 2024-04-20T02:51:22.825 INFO:tasks.rados.rados.0.smithi105.stdout:1037: seq_num 176 ranges {558032=719943,1849419=592048} 2024-04-20T02:51:22.831 INFO:tasks.rados.rados.0.smithi105.stdout:1037: writing smithi10548823-43 from 558032 to 1277975 tid 1 2024-04-20T02:51:22.838 INFO:tasks.rados.rados.0.smithi105.stdout:1037: writing smithi10548823-43 from 1849419 to 2441467 tid 2 2024-04-20T02:51:22.838 INFO:tasks.rados.rados.0.smithi105.stdout:1038: write oid 9 current snap is 14 2024-04-20T02:51:22.838 INFO:tasks.rados.rados.0.smithi105.stdout:1021: finishing write tid 4 to smithi10548823-11 2024-04-20T02:51:22.839 INFO:tasks.rados.rados.0.smithi105.stdout:1021: oid 11 updating version 0 to 70 2024-04-20T02:51:22.839 INFO:tasks.rados.rados.0.smithi105.stdout:1021: oid 11 version 70 is already newer than 68 2024-04-20T02:51:22.839 INFO:tasks.rados.rados.0.smithi105.stdout:1021: oid 11 version 70 is already newer than 69 2024-04-20T02:51:22.839 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 70 (ObjNum 168 snap 14 seq_num 168) dirty exists 2024-04-20T02:51:22.839 INFO:tasks.rados.rados.0.smithi105.stdout:1021: left oid 11 (ObjNum 168 snap 14 seq_num 168) 2024-04-20T02:51:22.839 INFO:tasks.rados.rados.0.smithi105.stdout:1025: finishing write tid 1 to smithi10548823-45 2024-04-20T02:51:22.839 INFO:tasks.rados.rados.0.smithi105.stdout:1022: finishing write tid 2 to smithi10548823-25 2024-04-20T02:51:22.839 INFO:tasks.rados.rados.0.smithi105.stdout:1022: finishing write tid 3 to smithi10548823-25 2024-04-20T02:51:22.839 INFO:tasks.rados.rados.0.smithi105.stdout:1038: seq_num 177 ranges {574839=795192,1907901=452059} 2024-04-20T02:51:22.845 INFO:tasks.rados.rados.0.smithi105.stdout:1038: writing smithi10548823-9 from 574839 to 1370031 tid 1 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1038: writing smithi10548823-9 from 1907901 to 2359960 tid 2 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1021: done (15 left) 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1039: snap_create 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1022: finishing write tid 4 to smithi10548823-25 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1022: finishing write tid 5 to smithi10548823-25 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1022: oid 25 updating version 0 to 84 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1022: oid 25 version 84 is already newer than 83 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1022: oid 25 updating version 84 to 85 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1022: oid 25 version 85 is already newer than 82 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 85 (ObjNum 169 snap 14 seq_num 169) dirty exists 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1022: left oid 25 (ObjNum 169 snap 14 seq_num 169) 2024-04-20T02:51:22.854 INFO:tasks.rados.rados.0.smithi105.stdout:1020: finishing write tid 3 to smithi10548823-12 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1020: finishing write tid 4 to smithi10548823-12 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1020: finishing write tid 5 to smithi10548823-12 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1020: oid 12 updating version 0 to 20 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1020: oid 12 version 20 is already newer than 19 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1020: oid 12 updating version 20 to 21 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1020: oid 12 updating version 21 to 22 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 22 (ObjNum 167 snap 14 seq_num 167) dirty exists 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1020: left oid 12 (ObjNum 167 snap 14 seq_num 167) 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1024: finishing write tid 1 to smithi10548823-44 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1025: finishing write tid 2 to smithi10548823-45 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1024: finishing write tid 2 to smithi10548823-44 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1023: finishing write tid 1 to smithi10548823-34 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1023: finishing write tid 2 to smithi10548823-34 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1025: finishing write tid 3 to smithi10548823-45 2024-04-20T02:51:22.855 INFO:tasks.rados.rados.0.smithi105.stdout:1024: finishing write tid 3 to smithi10548823-44 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1024: finishing write tid 4 to smithi10548823-44 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1024: finishing write tid 5 to smithi10548823-44 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1024: oid 44 updating version 0 to 46 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1024: oid 44 updating version 46 to 47 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1024: oid 44 updating version 47 to 48 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1024: oid 44 updating version 48 to 49 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 49 (ObjNum 171 snap 14 seq_num 171) dirty exists 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1024: left oid 44 (ObjNum 171 snap 14 seq_num 171) 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1025: finishing write tid 4 to smithi10548823-45 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1025: finishing write tid 5 to smithi10548823-45 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1025: oid 45 updating version 0 to 90 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1025: oid 45 updating version 90 to 92 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1025: oid 45 version 92 is already newer than 89 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:1025: oid 45 version 92 is already newer than 91 2024-04-20T02:51:22.856 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 92 (ObjNum 172 snap 14 seq_num 172) dirty exists 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1025: left oid 45 (ObjNum 172 snap 14 seq_num 172) 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1023: finishing write tid 3 to smithi10548823-34 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1023: finishing write tid 4 to smithi10548823-34 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1023: finishing write tid 5 to smithi10548823-34 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1023: oid 34 updating version 0 to 157 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1023: oid 34 version 157 is already newer than 156 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1023: oid 34 version 157 is already newer than 154 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1023: oid 34 version 157 is already newer than 155 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 157 (ObjNum 170 snap 14 seq_num 170) dirty exists 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1023: left oid 34 (ObjNum 170 snap 14 seq_num 170) 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1029: finishing write tid 1 to smithi10548823-18 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1030: got (0) Success 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 25 (ObjNum 129 snap 9 seq_num 129) dirty exists 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1029: finishing write tid 2 to smithi10548823-18 2024-04-20T02:51:22.857 INFO:tasks.rados.rados.0.smithi105.stdout:1029: finishing write tid 3 to smithi10548823-18 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1029: finishing write tid 4 to smithi10548823-18 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1029: oid 18 updating version 0 to 53 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1029: oid 18 version 53 is already newer than 51 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1029: oid 18 version 53 is already newer than 52 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 53 (ObjNum 173 snap 14 seq_num 173) dirty exists 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1029: left oid 18 (ObjNum 173 snap 14 seq_num 173) 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1031: finishing write tid 1 to smithi10548823-32 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1031: finishing write tid 2 to smithi10548823-32 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1032: finishing write tid 1 to smithi10548823-7 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1031: finishing write tid 3 to smithi10548823-32 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1032: finishing write tid 2 to smithi10548823-7 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1031: finishing write tid 4 to smithi10548823-32 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1032: finishing write tid 3 to smithi10548823-7 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1032: finishing write tid 4 to smithi10548823-7 2024-04-20T02:51:22.858 INFO:tasks.rados.rados.0.smithi105.stdout:1032: oid 7 updating version 0 to 160 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1032: oid 7 version 160 is already newer than 159 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1032: oid 7 updating version 160 to 161 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 161 (ObjNum 175 snap 14 seq_num 175) dirty exists 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1032: left oid 7 (ObjNum 175 snap 14 seq_num 175) 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1031: finishing write tid 5 to smithi10548823-32 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1031: oid 32 updating version 0 to 36 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1031: oid 32 updating version 36 to 38 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1031: oid 32 version 38 is already newer than 35 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1031: oid 32 version 38 is already newer than 37 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 38 (ObjNum 174 snap 14 seq_num 174) dirty exists 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1031: left oid 32 (ObjNum 174 snap 14 seq_num 174) 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1033: finishing set_chunk to oid 31 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 31 set_chunk returned error code -95 offset: 578048 length: 1196032 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 0 (ObjNum 142 snap 9 seq_num 142) dirty exists 2024-04-20T02:51:22.859 INFO:tasks.rados.rados.0.smithi105.stdout:1036: finishing set_chunk to oid 40 2024-04-20T02:51:22.860 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 156 (ObjNum 161 snap 13 seq_num 161) dirty exists 2024-04-20T02:51:22.861 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:22.861+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:22.870 INFO:tasks.rados.rados.0.smithi105.stdout:1037: finishing write tid 1 to smithi10548823-43 2024-04-20T02:51:22.873 INFO:tasks.rados.rados.0.smithi105.stdout:1037: finishing write tid 2 to smithi10548823-43 2024-04-20T02:51:22.873 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:22.874 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:22.874 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:22.874 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:22.874 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:22.874 INFO:tasks.rados.rados.0.smithi105.stdout:1037: finishing write tid 3 to smithi10548823-43 2024-04-20T02:51:22.874 INFO:tasks.rados.rados.0.smithi105.stdout:1037: finishing write tid 4 to smithi10548823-43 2024-04-20T02:51:22.875 INFO:tasks.rados.rados.0.smithi105.stdout:1037: oid 43 updating version 0 to 73 2024-04-20T02:51:22.875 INFO:tasks.rados.rados.0.smithi105.stdout:1037: oid 43 updating version 73 to 74 2024-04-20T02:51:22.875 INFO:tasks.rados.rados.0.smithi105.stdout:1037: oid 43 version 74 is already newer than 72 2024-04-20T02:51:22.875 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 74 (ObjNum 176 snap 14 seq_num 176) dirty exists 2024-04-20T02:51:22.875 INFO:tasks.rados.rados.0.smithi105.stdout:1037: left oid 43 (ObjNum 176 snap 14 seq_num 176) 2024-04-20T02:51:22.883 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:22.884 INFO:tasks.rados.rados.0.smithi105.stdout:1038: finishing write tid 1 to smithi10548823-9 2024-04-20T02:51:22.884 INFO:tasks.rados.rados.0.smithi105.stdout:1038: finishing write tid 2 to smithi10548823-9 2024-04-20T02:51:22.884 INFO:tasks.rados.rados.0.smithi105.stdout:1038: finishing write tid 3 to smithi10548823-9 2024-04-20T02:51:22.885 INFO:tasks.rados.rados.0.smithi105.stdout:1038: finishing write tid 4 to smithi10548823-9 2024-04-20T02:51:22.885 INFO:tasks.rados.rados.0.smithi105.stdout:1038: oid 9 updating version 0 to 160 2024-04-20T02:51:22.885 INFO:tasks.rados.rados.0.smithi105.stdout:1038: oid 9 updating version 160 to 161 2024-04-20T02:51:22.885 INFO:tasks.rados.rados.0.smithi105.stdout:1038: oid 9 version 161 is already newer than 159 2024-04-20T02:51:22.885 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 161 (ObjNum 177 snap 14 seq_num 177) dirty exists 2024-04-20T02:51:22.885 INFO:tasks.rados.rados.0.smithi105.stdout:1038: left oid 9 (ObjNum 177 snap 14 seq_num 177) 2024-04-20T02:51:22.968 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:22.969+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:23.054 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:23.054 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 63.449893213000003, 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.055 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:23.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:51:23.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:51:23.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:51:23.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:51:23.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.061 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:30 4.8 4:1c68e542:::smithi10548823-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.689741025000004, 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033039351000000002, 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 30 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364066+0000", 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364312+0000", 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000246708 2024-04-20T02:51:23.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364328+0000", 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5376999999999999e-05 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364339+0000", 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.092e-05 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375692+0000", 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011353103999999999 2024-04-20T02:51:23.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375805+0000", 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000113414 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375970+0000", 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016508399999999999 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388615+0000", 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012645145 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391156+0000", 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025406410000000002 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396990+0000", 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0058340390000000001 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397042+0000", 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2454000000000003e-05 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397106+0000", 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3632000000000003e-05 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:31 4.8 4:1c68e542:::smithi10548823-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.689454329, 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032910542000000001, 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.067 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 31 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364357+0000", 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1160000000000001e-06 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364363+0000", 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1680000000000001e-06 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364367+0000", 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8750000000000002e-06 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388814+0000", 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024447437999999998 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388914+0000", 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9557999999999998e-05 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389062+0000", 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014799600000000001 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391160+0000", 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020979760000000001 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396038+0000", 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048784689999999999 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397155+0000", 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001116286 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397208+0000", 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3273000000000003e-05 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397264+0000", 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5962999999999999e-05 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:32 4.8 4:1c68e542:::smithi10548823-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.689435615000001, 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033121840999999999, 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 32 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364375+0000", 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1149999999999998e-06 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364382+0000", 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4259999999999997e-06 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364387+0000", 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6129999999999997e-06 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389813+0000", 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025426621 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389905+0000", 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2011999999999999e-05 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397299+0000", 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007393897 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397395+0000", 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5293999999999999e-05 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397494+0000", 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9289000000000004e-05 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:23.077 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.686026814000002, 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:23.081 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.572723773, 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:66 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.567365885000001, 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033365526, 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.087 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 66 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486438+0000", 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486778+0000", 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000339709 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486798+0000", 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0462000000000002e-05 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486812+0000", 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3589e-05 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496889+0000", 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010077477 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496978+0000", 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8205e-05 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.497125+0000", 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000147665 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518444+0000", 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.021318713 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519407+0000", 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00096312999999999998 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519727+0000", 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031953500000000002 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519764+0000", 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7406000000000001e-05 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519807+0000", 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3402000000000002e-05 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.092 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:67 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2268912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.566980700999999, 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033307638000000001, 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 67 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486826+0000", 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486831+0000", 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0189999999999997e-06 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486839+0000", 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4710000000000001e-06 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486844+0000", 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.57e-06 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509240+0000", 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022395241 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509338+0000", 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8034000000000004e-05 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509498+0000", 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015993999999999999 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518471+0000", 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0089732730000000004 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519859+0000", 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013877130000000001 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520042+0000", 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182981 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520084+0000", 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2518999999999997e-05 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520135+0000", 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0491000000000002e-05 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:23.098 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.566958255999999, 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.212938121000001, 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.103 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:51:23.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:51:23.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.202574515999999, 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.107 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:51:23.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:51:23.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:51:23.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.194490037000001, 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.111 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:51:23.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:51:23.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:51:23.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:981 4.7 e32/27 4:e9876a63:::smithi10548823-30:head v 32'19, mlcod=30'17)", 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.464437562999997, 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033092985999999998, 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.115 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589369+0000", 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590134+0000", 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076479 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590148+0000", 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3679999999999999e-05 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590157+0000", 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:23.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590209+0000", 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2237999999999997e-05 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590235+0000", 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6251e-05 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622430+0000", 2024-04-20T02:51:23.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032194953999999998 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622463+0000", 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2780000000000001e-05 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1230 4.5 e38/22 4:a94dc531:::smithi10548823-32:head v 38'11, mlcod=29'8)", 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.641292164999999, 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033312785999999997, 2024-04-20T02:51:23.119 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412515+0000", 2024-04-20T02:51:23.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412827+0000", 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031200999999999999 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412835+0000", 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.61e-06 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412842+0000", 2024-04-20T02:51:23.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6990000000000003e-06 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412921+0000", 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8380000000000005e-05 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412944+0000", 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3609e-05 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445795+0000", 2024-04-20T02:51:23.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032850279000000003 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445828+0000", 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3673000000000001e-05 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.912931812, 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:51:23.123 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:51:23.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:51:23.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:51:23.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:51:23.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.907794702, 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:51:23.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:51:23.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:51:23.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:51:23.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:51:23.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:51:23.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.135 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.907521689999999, 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:23.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:51:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:51:23.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.907492305000002, 2024-04-20T02:51:23.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:51:23.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.598598340000001, 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035060916999999997, 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:23.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:23.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456603+0000", 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001394295 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456616+0000", 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2605999999999999e-05 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456631+0000", 2024-04-20T02:51:23.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4735e-05 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456682+0000", 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1048000000000003e-05 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456705+0000", 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2881e-05 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490235+0000", 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033530250999999997 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490270+0000", 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5379000000000001e-05 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.152 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.220165006, 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.156 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.157 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:23.158 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:23.158 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:23.159+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:23.258 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:23.259+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:23.329 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:23.330 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:23.330 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:23.339 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:23.383 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:23.384+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:23.461 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:23.461+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:23.468 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:23.468 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:23.468 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:23.468 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:23.468 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:23.478 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:23.554 INFO:tasks.rados.rados.0.smithi105.stdout:1020: done (15 left) 2024-04-20T02:51:23.554 INFO:tasks.rados.rados.0.smithi105.stdout:1022: done (14 left) 2024-04-20T02:51:23.554 INFO:tasks.rados.rados.0.smithi105.stdout:1023: done (13 left) 2024-04-20T02:51:23.554 INFO:tasks.rados.rados.0.smithi105.stdout:1024: done (12 left) 2024-04-20T02:51:23.554 INFO:tasks.rados.rados.0.smithi105.stdout:1025: done (11 left) 2024-04-20T02:51:23.554 INFO:tasks.rados.rados.0.smithi105.stdout:1029: done (10 left) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1030: done (9 left) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1031: done (8 left) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1032: done (7 left) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1033: done (6 left) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1035: done (5 left) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1036: done (4 left) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1037: done (3 left) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1038: done (2 left) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1039: done (1 left) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1040: read oid 31 snap 13 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1040: expect (ObjNum 142 snap 9 seq_num 142) 2024-04-20T02:51:23.555 INFO:tasks.rados.rados.0.smithi105.stdout:1041: read oid 19 snap 11 2024-04-20T02:51:23.557 INFO:tasks.rados.rados.0.smithi105.stdout:1041: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:23.557 INFO:tasks.rados.rados.0.smithi105.stdout:1042: rollback oid 23 current snap is 15 2024-04-20T02:51:23.557 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 23 to 6 2024-04-20T02:51:23.565 INFO:tasks.rados.rados.0.smithi105.stdout:1043: write (excl) oid 36 current snap is 15 2024-04-20T02:51:23.565 INFO:tasks.rados.rados.0.smithi105.stdout:1043: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-04-20T02:51:23.566 INFO:tasks.rados.rados.0.smithi105.stdout:1043: writing smithi10548823-36 from 591646 to 1062087 tid 1 2024-04-20T02:51:23.573 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:23.566+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:23.574 INFO:tasks.rados.rados.0.smithi105.stdout:1043: writing smithi10548823-36 from 1566383 to 2162100 tid 2 2024-04-20T02:51:23.574 INFO:tasks.rados.rados.0.smithi105.stdout:1043: writing smithi10548823-36 from 2888822 to 2991646 tid 3 2024-04-20T02:51:23.574 INFO:tasks.rados.rados.0.smithi105.stdout:1044: read oid 41 snap 13 2024-04-20T02:51:23.574 INFO:tasks.rados.rados.0.smithi105.stdout:1044: expect (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:51:23.574 INFO:tasks.rados.rados.0.smithi105.stdout:1045: tier_promote oid 38 2024-04-20T02:51:23.574 INFO:tasks.rados.rados.0.smithi105.stdout:1034: expect (ObjNum 144 snap 9 seq_num 144) 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:152 3.2 e48/47 3:54f0c00e:::2528189ec1c891f179c539fa23af2c87b9e8d2c19f70e70fac8934d3cc51ccee:head v 48'96, mlcod=0'0)", 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.227588320999999, 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020196797999999998, 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433091+0000", 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000185589 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433102+0000", 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433108+0000", 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5660000000000003e-06 2024-04-20T02:51:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439137+0000", 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060284329999999997 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439166+0000", 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9278e-05 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453075+0000", 2024-04-20T02:51:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013909163 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453103+0000", 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7725999999999998e-05 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:154 3.2 e48/47 3:7d72d9ce:::7b4ada8765f224bb4a516820a83f51a402b4ee671c5f45bb3fbe5e6656cf1f14:head v 48'97, mlcod=42'91)", 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.223682689, 2024-04-20T02:51:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016320239, 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436811+0000", 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437600+0000", 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078939699999999995 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437613+0000", 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2588e-05 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437624+0000", 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0542e-05 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442541+0000", 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049178809999999998 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442565+0000", 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3558e-05 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453110+0000", 2024-04-20T02:51:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010545172 2024-04-20T02:51:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453132+0000", 2024-04-20T02:51:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1929e-05 2024-04-20T02:51:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:155 3.2 e48/47 3:715fab8c:::b945a2927747be608263bdbe1fe814f665c73382cb3375555da5aa95a92567ff:head v 48'99, mlcod=42'91)", 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.220895634000001, 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017321785999999999, 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439598+0000", 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439954+0000", 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00035586299999999999 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439970+0000", 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5889e-05 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439981+0000", 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0754000000000001e-05 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443643+0000", 2024-04-20T02:51:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036621259999999999 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443671+0000", 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8289e-05 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456885+0000", 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013213964999999999 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456921+0000", 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1635 3.2 e51/49 3:67667c16:::7aea211498f946c134ed12b1f5632511a2f083a0eb2114d1051f38dc7326b113:head v 51'100, mlcod=0'0)", 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.813004686999999, 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019392829, 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847489+0000", 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850819+0000", 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033292069999999998 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850831+0000", 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2320000000000001e-05 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850838+0000", 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2980000000000003e-06 2024-04-20T02:51:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850874+0000", 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5800999999999998e-05 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850890+0000", 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5536000000000002e-05 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866849+0000", 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015959346999999999 2024-04-20T02:51:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866883+0000", 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3776999999999999e-05 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.211856448000001, 2024-04-20T02:51:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1665 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'78, mlcod=50'73)", 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.137256595, 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025918594, 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523236+0000", 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523658+0000", 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000422082 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523676+0000", 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7716000000000001e-05 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523686+0000", 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9790000000000005e-06 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523734+0000", 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7579e-05 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523759+0000", 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5593e-05 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549122+0000", 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025363087999999999 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549157+0000", 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4242999999999997e-05 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:51:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.205348216000001, 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034212370999999998, 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456260+0000", 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011140480000000001 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456273+0000", 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456280+0000", 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1130000000000003e-06 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456331+0000", 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1446999999999997e-05 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456357+0000", 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6001999999999999e-05 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489309+0000", 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032951972000000003 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489359+0000", 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9552000000000003e-05 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1775 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'41, mlcod=53'38)", 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.9069667260000003, 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038672274999999999, 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754868+0000", 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013404619999999999 2024-04-20T02:51:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754881+0000", 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.323e-05 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754892+0000", 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1154e-05 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754967+0000", 2024-04-20T02:51:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4326999999999995e-05 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.755006+0000", 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9338000000000002e-05 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792154+0000", 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037147924999999998 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792200+0000", 2024-04-20T02:51:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6044999999999997e-05 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1776 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'42, mlcod=53'38)", 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.8869865479999994, 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019186657999999999, 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773507+0000", 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774271+0000", 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076432799999999999 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774284+0000", 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2483000000000001e-05 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774294+0000", 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6569999999999992e-06 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774338+0000", 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4576000000000001e-05 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774358+0000", 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9449999999999998e-05 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792664+0000", 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018306413000000001 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792695+0000", 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0685000000000002e-05 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1822 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'94, mlcod=55'92)", 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5569445930000008, 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031823907999999998, 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103786+0000", 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000236658 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103797+0000", 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0951000000000001e-05 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103804+0000", 2024-04-20T02:51:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0960000000000003e-06 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103860+0000", 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5781e-05 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103887+0000", 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6999999999999999e-05 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135333+0000", 2024-04-20T02:51:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031446004 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135374+0000", 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0778999999999997e-05 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1823 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'95, mlcod=55'92)", 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5424575869999995, 2024-04-20T02:51:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017980995999999999, 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118036+0000", 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118363+0000", 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00032734600000000001 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118381+0000", 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7816e-05 2024-04-20T02:51:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118391+0000", 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0155e-05 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119591+0000", 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011995560000000001 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119618+0000", 2024-04-20T02:51:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7475e-05 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135988+0000", 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016370096000000001 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.136018+0000", 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9745999999999999e-05 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1836 4.1 e58/46 4:8b927dd6:::smithi10548823-28:head v 58'99, mlcod=56'96)", 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.174928585, 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030374581000000001, 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485930+0000", 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00036455800000000002 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485941+0000", 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1017e-05 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485950+0000", 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.85e-06 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486000+0000", 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9589999999999998e-05 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486025+0000", 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.529e-05 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515909+0000", 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029884131000000001 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515941+0000", 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1569000000000002e-05 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1837 4.1 e58/46 4:8b927dd6:::smithi10548823-28:head v 58'100, mlcod=56'96)", 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:15.501011+0000", 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.1594833310000006, 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015450057999999999, 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501011+0000", 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501011+0000", 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501010+0000", 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501412+0000", 2024-04-20T02:51:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040223499999999997 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501426+0000", 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3801999999999999e-05 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501435+0000", 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7970000000000006e-06 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501480+0000", 2024-04-20T02:51:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5695999999999998e-05 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.501505+0000", 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4213999999999999e-05 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.516435+0000", 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014930047 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.516461+0000", 2024-04-20T02:51:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6571999999999999e-05 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1616736809999999, 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:51:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1616082319999999, 2024-04-20T02:51:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1615827339999996, 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1891 4.17 e61/60 4:e9876a63:::smithi10548823-30:head v 61'45, mlcod=0'0)", 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1470369380000003, 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018777245000000001, 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513457+0000", 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514596+0000", 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011393289999999999 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514612+0000", 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6311e-05 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514623+0000", 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056999999999999e-05 2024-04-20T02:51:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514670+0000", 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6557999999999997e-05 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514698+0000", 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8121000000000001e-05 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.532204+0000", 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017505969999999999 2024-04-20T02:51:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.532235+0000", 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0731000000000001e-05 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1892 4.17 e61/60 4:e9876a63:::smithi10548823-30:head v 61'46, mlcod=0'0)", 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1236979539999998, 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020303921999999999, 2024-04-20T02:51:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536796+0000", 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537578+0000", 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078217499999999997 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537592+0000", 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3721e-05 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537602+0000", 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0380000000000001e-05 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537638+0000", 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5559000000000003e-05 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537657+0000", 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9242e-05 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.557064+0000", 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019406970999999999 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.557101+0000", 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6572000000000002e-05 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2004 4.13 e65/63 4:cfd1e4d4:::smithi10548823-9:head v 65'159, mlcod=64'156)", 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.81235768600000002, 2024-04-20T02:51:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036995865000000003, 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849186+0000", 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010492889999999999 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849197+0000", 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1365999999999999e-05 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849204+0000", 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1069999999999999e-06 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849250+0000", 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5213999999999998e-05 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849279+0000", 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9149000000000002e-05 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885093+0000", 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035813921999999998 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885133+0000", 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0077999999999998e-05 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2005 4.13 e65/63 4:cfd1e4d4:::smithi10548823-9:head v 65'160, mlcod=64'156)", 2024-04-20T02:51:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.79249671899999996, 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.01736478, 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867997+0000", 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868547+0000", 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000550287 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868563+0000", 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5676e-05 2024-04-20T02:51:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868572+0000", 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0440000000000001e-06 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.874844+0000", 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0062724460000000001 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.874865+0000", 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0712000000000001e-05 2024-04-20T02:51:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885334+0000", 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010468691 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885363+0000", 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8923000000000001e-05 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:23.746 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:23.747 INFO:tasks.rados.rados.0.smithi105.stdout:1042: finishing rollback tid 0 to smithi10548823-23 2024-04-20T02:51:23.747 INFO:tasks.rados.rados.0.smithi105.stdout:1042: finishing rollback tid 1 to smithi10548823-23 2024-04-20T02:51:23.747 INFO:tasks.rados.rados.0.smithi105.stdout:1042: finishing rollback tid 2 to smithi10548823-23 2024-04-20T02:51:23.747 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 61 (ObjNum 67 snap 0 seq_num 67) dirty exists 2024-04-20T02:51:23.747 INFO:tasks.rados.rados.0.smithi105.stdout:1040: expect (ObjNum 142 snap 9 seq_num 142) 2024-04-20T02:51:23.747 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:23.747+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:23.748 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:23.840 INFO:tasks.rados.rados.0.smithi105.stdout:1045: got (0) Success 2024-04-20T02:51:23.840 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 26 (ObjNum 97 snap 0 seq_num 231287160) clean dne 2024-04-20T02:51:23.840 INFO:tasks.rados.rados.0.smithi105.stdout:1044: expect (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:51:23.848 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:23.848+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:23.878 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:23.878 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:23.878 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:23.878 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:23.888 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:23.948 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:23.949+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:23.990 INFO:tasks.rados.rados.0.smithi105.stdout:1041: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:24.050 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:24.050+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:24.057 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:24.058 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:24.058 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:24.058 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:24.058 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:24.068 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:24.110 INFO:tasks.rados.rados.0.smithi105.stdout:1043: finishing write tid 1 to smithi10548823-36 2024-04-20T02:51:24.110 INFO:tasks.rados.rados.0.smithi105.stdout:1043: finishing write tid 2 to smithi10548823-36 2024-04-20T02:51:24.110 INFO:tasks.rados.rados.0.smithi105.stdout:1043: finishing write tid 3 to smithi10548823-36 2024-04-20T02:51:24.110 INFO:tasks.rados.rados.0.smithi105.stdout:1034: done (6 left) 2024-04-20T02:51:24.110 INFO:tasks.rados.rados.0.smithi105.stdout:1040: done (5 left) 2024-04-20T02:51:24.110 INFO:tasks.rados.rados.0.smithi105.stdout:1041: done (4 left) 2024-04-20T02:51:24.110 INFO:tasks.rados.rados.0.smithi105.stdout:1042: done (3 left) 2024-04-20T02:51:24.110 INFO:tasks.rados.rados.0.smithi105.stdout:1044: done (2 left) 2024-04-20T02:51:24.110 INFO:tasks.rados.rados.0.smithi105.stdout:1045: done (1 left) 2024-04-20T02:51:24.110 INFO:tasks.rados.rados.0.smithi105.stdout:1046: delete oid 5 current snap is 15 2024-04-20T02:51:24.111 INFO:tasks.rados.rados.0.smithi105.stdout:1043: finishing write tid 4 to smithi10548823-36 2024-04-20T02:51:24.111 INFO:tasks.rados.rados.0.smithi105.stdout:1043: finishing write tid 5 to smithi10548823-36 2024-04-20T02:51:24.111 INFO:tasks.rados.rados.0.smithi105.stdout:1043: oid 36 updating version 0 to 161 2024-04-20T02:51:24.111 INFO:tasks.rados.rados.0.smithi105.stdout:1043: oid 36 updating version 161 to 162 2024-04-20T02:51:24.111 INFO:tasks.rados.rados.0.smithi105.stdout:1043: oid 36 version 162 is already newer than 159 2024-04-20T02:51:24.111 INFO:tasks.rados.rados.0.smithi105.stdout:1043: oid 36 version 162 is already newer than 160 2024-04-20T02:51:24.111 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 162 (ObjNum 178 snap 15 seq_num 178) dirty exists 2024-04-20T02:51:24.111 INFO:tasks.rados.rados.0.smithi105.stdout:1043: left oid 36 (ObjNum 178 snap 15 seq_num 178) 2024-04-20T02:51:24.111 INFO:tasks.rados.rados.0.smithi105.stdout:1043: done (1 left) 2024-04-20T02:51:24.111 INFO:tasks.rados.rados.0.smithi105.stdout:1046: done (0 left) 2024-04-20T02:51:24.112 INFO:tasks.rados.rados.0.smithi105.stdout:1047: set_chunk oid 23 target oid 2024-04-20T02:51:24.151 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:24.151+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:24.154 INFO:tasks.rados.rados.0.smithi105.stdout:1047: set_chunk oid 23 offset: 473088 length: 1714688 target oid b13b8a8d7724a2b6951a53e14e5e568e1aa45f9c18b0373ab4a92ed71f345848 offset: 0 snap -1 2024-04-20T02:51:24.154 INFO:tasks.rados.rados.0.smithi105.stdout:1048: set_chunk oid 45 target oid 2024-04-20T02:51:24.167 INFO:tasks.rados.rados.0.smithi105.stdout:1048: set_chunk oid 45 offset: 2249728 length: 332800 target oid ce85a86357c90c981bb52c7d7c4656989fd90dba277956a79f89a712c7009ab4 offset: 0 snap 9 2024-04-20T02:51:24.168 INFO:tasks.rados.rados.0.smithi105.stdout:1049: read oid 25 snap -1 2024-04-20T02:51:24.168 INFO:tasks.rados.rados.0.smithi105.stdout:1049: expect (ObjNum 169 snap 14 seq_num 169) 2024-04-20T02:51:24.168 INFO:tasks.rados.rados.0.smithi105.stdout:1050: snap_remove snap 14 2024-04-20T02:51:24.214 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:24.214 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:24.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:24.214 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:24.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.215 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:927 4.3 4:cf11ccc0:::smithi10548823-4:head [write 3121996~104915] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:24.215 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:24.215 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.835801365999998, 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058986447999999997, 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 927 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377548+0000", 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377607+0000", 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9413999999999997e-05 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377616+0000", 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4880000000000007e-06 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377621+0000", 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.682e-06 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407778+0000", 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030156110999999999 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407866+0000", 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8645000000000003e-05 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407985+0000", 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118874 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411741+0000", 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037563060000000001 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411854+0000", 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112898 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436438+0000", 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024583885 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436483+0000", 2024-04-20T02:51:24.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5334999999999999e-05 2024-04-20T02:51:24.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436535+0000", 2024-04-20T02:51:24.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1780000000000002e-05 2024-04-20T02:51:24.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.222 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.222 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:928 4.3 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (55) in=7b,truncate 3226911] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:24.222 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.835722760000003, 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059064512999999999, 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 928 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:24.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377630+0000", 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7259999999999998e-06 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377633+0000", 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1489999999999998e-06 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377636+0000", 2024-04-20T02:51:24.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7760000000000002e-06 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410754+0000", 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033117944000000003 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410850+0000", 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6522999999999997e-05 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411008+0000", 2024-04-20T02:51:24.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158157 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436151+0000", 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025142813 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436160+0000", 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1460000000000002e-06 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436588+0000", 2024-04-20T02:51:24.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00042776499999999998 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436631+0000", 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3084000000000003e-05 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436692+0000", 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0829000000000003e-05 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:929 4.3 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.835711619000001, 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059214058, 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 929 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377638+0000", 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377640+0000", 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5430000000000001e-06 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377643+0000", 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0900000000000001e-06 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377647+0000", 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2230000000000001e-06 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411271+0000", 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033623635999999998 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411329+0000", 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8043e-05 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436728+0000", 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025399300999999999 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436793+0000", 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5041000000000003e-05 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436853+0000", 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9446e-05 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.233 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:936 4.3 4:ca036f62:::smithi10548823-34:head [write 443718~615603] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.818676285999999, 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060880228000000002, 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 936 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:24.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:24.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394672+0000", 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394825+0000", 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015343299999999999 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394844+0000", 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8414999999999999e-05 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394859+0000", 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4738e-05 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411879+0000", 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017020469999999999 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411954+0000", 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5239000000000002e-05 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412104+0000", 2024-04-20T02:51:24.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014944399999999999 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436402+0000", 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024297828 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436409+0000", 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1779999999999997e-06 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455390+0000", 2024-04-20T02:51:24.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018981221 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455508+0000", 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117556 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455554+0000", 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6570000000000003e-05 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:937 4.3 4:ca036f62:::smithi10548823-34:head [write 1564138~470341] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.813337955999998, 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060119655000000001, 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 937 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400010+0000", 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400222+0000", 2024-04-20T02:51:24.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000211208 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400238+0000", 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419e-05 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400250+0000", 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2091e-05 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425851+0000", 2024-04-20T02:51:24.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025600795999999999 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425965+0000", 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011411799999999999 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426134+0000", 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168517 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436937+0000", 2024-04-20T02:51:24.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01080328 2024-04-20T02:51:24.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455351+0000", 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018414607 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460001+0000", 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046491809999999996 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460086+0000", 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5066000000000002e-05 2024-04-20T02:51:24.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460132+0000", 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6174999999999998e-05 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:938 4.3 4:ca036f62:::smithi10548823-34:head [write 2633066~610652] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.806572500000001, 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054360654000000001, 2024-04-20T02:51:24.246 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 938 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406775+0000", 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406947+0000", 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171619 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406962+0000", 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5248999999999999e-05 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406979+0000", 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7334999999999999e-05 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436954+0000", 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029974218 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437007+0000", 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3553e-05 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437110+0000", 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455605+0000", 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018494295000000001 2024-04-20T02:51:24.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455829+0000", 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022443100000000001 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461044+0000", 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052150499999999997 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461090+0000", 2024-04-20T02:51:24.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6100999999999998e-05 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461138+0000", 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8003000000000001e-05 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:939 4.3 4:ca036f62:::smithi10548823-34:head [setxattr _header (56) in=7b,truncate 3243718] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.806360189000003, 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054380381999999998, 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.252 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 939 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406989+0000", 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406993+0000", 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9400000000000004e-06 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406999+0000", 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4389999999999999e-06 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407008+0000", 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3190000000000001e-06 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.454982+0000", 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047973729999999999 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455051+0000", 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9197999999999997e-05 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455157+0000", 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106302 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456120+0000", 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096236599999999998 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456272+0000", 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000152573 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461274+0000", 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005002055 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461318+0000", 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3282999999999999e-05 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461370+0000", 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2704999999999998e-05 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:940 4.3 4:ca036f62:::smithi10548823-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.80633564, 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054543401999999998, 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:24.258 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 940 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:24.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407014+0000", 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407017+0000", 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9790000000000002e-06 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407022+0000", 2024-04-20T02:51:24.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5539999999999997e-06 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407027+0000", 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6099999999999999e-06 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455643+0000", 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.048616331999999998 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455714+0000", 2024-04-20T02:51:24.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0610000000000003e-05 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461403+0000", 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056888219999999996 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461480+0000", 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7342000000000006e-05 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461558+0000", 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7780000000000004e-05 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.668690105000003, 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:51:24.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.280767644000001, 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.268 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.273 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.280507438000001, 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:51:24.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:51:24.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:51:24.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:51:24.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:51:24.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.280485970000001, 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:24.284 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.086814038, 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:51:24.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:51:24.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:133 3.1 3:8efdf53d:::0188443c5b8894a8247535d52a547bece3a4c5f490fb220852b80e77d71ddcda:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.086089516000001, 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060974049000000002, 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.290 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127260+0000", 2024-04-20T02:51:24.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127435+0000", 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017459800000000001 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127443+0000", 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3790000000000006e-06 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127450+0000", 2024-04-20T02:51:24.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5089999999999999e-06 2024-04-20T02:51:24.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.129973+0000", 2024-04-20T02:51:24.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025229639999999999 2024-04-20T02:51:24.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130090+0000", 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117521 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130323+0000", 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023271199999999999 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.147581+0000", 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017258440999999999 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155339+0000", 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077577189999999997 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188139+0000", 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032800425000000001 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188195+0000", 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5773999999999998e-05 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188235+0000", 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9427e-05 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:136 3.1 3:a965252c:::a490375a0954219328244dfa32cf9967dfac8ed04667cb9b5ce15f4910db2ee6:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.085894299, 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060878320999999999, 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127455+0000", 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128197+0000", 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074176999999999999 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128208+0000", 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0498e-05 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128216+0000", 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2630000000000003e-06 2024-04-20T02:51:24.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135797+0000", 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0075808669999999998 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135930+0000", 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013299000000000001 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.136192+0000", 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026213899999999999 2024-04-20T02:51:24.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148053+0000", 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011860723 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155346+0000", 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0072930039999999996 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188270+0000", 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032924299999999997 2024-04-20T02:51:24.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188298+0000", 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8041000000000001e-05 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188334+0000", 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6247e-05 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.301 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:139 3.1 3:90772484:::f9b8a83c02fab1c87f6aef9574ab68b1d6a75b657e100bac3693f85cac340cb2:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.079236374000001, 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054361675999999998, 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134113+0000", 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134645+0000", 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053281300000000004 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134662+0000", 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6503000000000001e-05 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134672+0000", 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8090000000000004e-06 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148073+0000", 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013401091 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148177+0000", 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104344 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148417+0000", 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023968899999999999 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155385+0000", 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0069679260000000002 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155725+0000", 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034046800000000002 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188383+0000", 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032658154000000002 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188433+0000", 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0059999999999998e-05 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188475+0000", 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2049999999999999e-05 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:140 3.1 3:b80bf598:::82930be3d6fd0d013eae1f4ab54a437986489fd06a4f4934c2432c14687252ab:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.078671118999999, 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053882434, 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134678+0000", 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134756+0000", 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7527999999999994e-05 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134763+0000", 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.092e-06 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0290000000000002e-06 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152310+0000", 2024-04-20T02:51:24.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017540581999999999 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152431+0000", 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012166 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152621+0000", 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000189944 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155992+0000", 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033703470000000001 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160642+0000", 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046505239999999996 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188508+0000", 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027866134000000001 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188534+0000", 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5945999999999999e-05 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188561+0000", 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7300999999999998e-05 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1799 4.6 4:697da67d:::smithi10548823-48:head [write 588278~715406,stat] snapc 13=[13,d] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:24.313 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.139745963999999, 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053216977999999998, 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1799 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073603+0000", 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073742+0000", 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000138727 2024-04-20T02:51:24.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073754+0000", 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1970000000000001e-05 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073761+0000", 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.883e-06 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073806+0000", 2024-04-20T02:51:24.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5840999999999997e-05 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073872+0000", 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5520999999999998e-05 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074005+0000", 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013345499999999999 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124313+0000", 2024-04-20T02:51:24.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050307628 2024-04-20T02:51:24.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124361+0000", 2024-04-20T02:51:24.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7608999999999998e-05 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126705+0000", 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0023443790000000002 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126770+0000", 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4737e-05 2024-04-20T02:51:24.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126821+0000", 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1375000000000002e-05 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1801 4.6 4:697da67d:::smithi10548823-48:head [setxattr _header (57) in=7b,truncate 2588278] snapc 13=[13,d] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.133571011000001, 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053362723000000001, 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.320 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1801 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079782+0000", 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7840000000000001e-06 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079790+0000", 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3270000000000003e-06 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079799+0000", 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5040000000000008e-06 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123583+0000", 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043783913000000001 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123714+0000", 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013130699999999999 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123889+0000", 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017436200000000001 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132488+0000", 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085989889999999996 2024-04-20T02:51:24.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132702+0000", 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021448600000000001 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133026+0000", 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032330500000000002 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133079+0000", 2024-04-20T02:51:24.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3127000000000001e-05 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133142+0000", 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3150000000000004e-05 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1802 4.6 4:697da67d:::smithi10548823-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.133537326000001, 2024-04-20T02:51:24.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053694407999999999, 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1802 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079812+0000", 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079815+0000", 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1329999999999999e-06 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079823+0000", 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.401e-06 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079828+0000", 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4149999999999998e-06 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124387+0000", 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044558788000000002 2024-04-20T02:51:24.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124468+0000", 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1447999999999997e-05 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133186+0000", 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0087175610000000004 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133344+0000", 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158393 2024-04-20T02:51:24.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:24.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:24.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:24.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133507+0000", 2024-04-20T02:51:24.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000162829 2024-04-20T02:51:24.331 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.331 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.331 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.331 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:24.331 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:24.331 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:24.333 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:24.333 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:24.332+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:24.433 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:24.434+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:24.502 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:24.502 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:24.502 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:24.502 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:24.512 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:24.535 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:24.535+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:24.571 INFO:tasks.rados.rados.0.smithi105.stdout:1050: done (3 left) 2024-04-20T02:51:24.571 INFO:tasks.rados.rados.0.smithi105.stdout:1051: write (excl) oid 24 current snap is 15 2024-04-20T02:51:24.572 INFO:tasks.rados.rados.0.smithi105.stdout:1051: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-04-20T02:51:24.576 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:51:24.578 INFO:tasks.rados.rados.0.smithi105.stdout:1051: writing smithi10548823-24 from 625260 to 1246199 tid 1 2024-04-20T02:51:24.590 INFO:tasks.rados.rados.0.smithi105.stdout:1051: writing smithi10548823-24 from 1683347 to 2282733 tid 2 2024-04-20T02:51:24.590 INFO:tasks.rados.rados.0.smithi105.stdout:1051: writing smithi10548823-24 from 2943668 to 3025260 tid 3 2024-04-20T02:51:24.590 INFO:tasks.rados.rados.0.smithi105.stdout:1052: rollback oid 15 current snap is 15 2024-04-20T02:51:24.590 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 15 to 11 2024-04-20T02:51:24.590 INFO:tasks.rados.rados.0.smithi105.stdout:1047: finishing set_chunk to oid 23 2024-04-20T02:51:24.590 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 23 set_chunk returned error code -95 offset: 473088 length: 1714688 2024-04-20T02:51:24.590 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 0 (ObjNum 67 snap 0 seq_num 67) dirty exists 2024-04-20T02:51:24.591 INFO:tasks.rados.rados.0.smithi105.stdout:1048: finishing set_chunk to oid 45 2024-04-20T02:51:24.591 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 2249728 length: 332800 2024-04-20T02:51:24.591 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 0 (ObjNum 172 snap 14 seq_num 172) dirty exists 2024-04-20T02:51:24.591 INFO:tasks.rados.rados.0.smithi105.stdout:1049: expect (ObjNum 169 snap 14 seq_num 169) 2024-04-20T02:51:24.635 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:24.636+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:24.679 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:24.704 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:24.705 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:24.705 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:24.705 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:24.705 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:24.716 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:24.736 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:24.736+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1051: finishing write tid 1 to smithi10548823-24 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1051: finishing write tid 2 to smithi10548823-24 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1051: finishing write tid 3 to smithi10548823-24 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1051: finishing write tid 4 to smithi10548823-24 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1051: finishing write tid 5 to smithi10548823-24 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1051: oid 24 updating version 0 to 161 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1051: oid 24 version 161 is already newer than 159 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1051: oid 24 version 161 is already newer than 160 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1051: oid 24 updating version 161 to 162 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 162 (ObjNum 180 snap 15 seq_num 180) dirty exists 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1051: left oid 24 (ObjNum 180 snap 15 seq_num 180) 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1052: finishing rollback tid 1 to smithi10548823-15 2024-04-20T02:51:24.752 INFO:tasks.rados.rados.0.smithi105.stdout:1052: finishing rollback tid 2 to smithi10548823-15 2024-04-20T02:51:24.753 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 164 (ObjNum 157 snap 11 seq_num 157) dirty exists 2024-04-20T02:51:24.753 INFO:tasks.rados.rados.0.smithi105.stdout:1047: done (4 left) 2024-04-20T02:51:24.753 INFO:tasks.rados.rados.0.smithi105.stdout:1048: done (3 left) 2024-04-20T02:51:24.753 INFO:tasks.rados.rados.0.smithi105.stdout:1049: done (2 left) 2024-04-20T02:51:24.753 INFO:tasks.rados.rados.0.smithi105.stdout:1051: done (1 left) 2024-04-20T02:51:24.753 INFO:tasks.rados.rados.0.smithi105.stdout:1052: done (0 left) 2024-04-20T02:51:24.753 INFO:tasks.rados.rados.0.smithi105.stdout:1053: write oid 28 current snap is 15 2024-04-20T02:51:24.753 INFO:tasks.rados.rados.0.smithi105.stdout:1053: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-04-20T02:51:24.759 INFO:tasks.rados.rados.0.smithi105.stdout:1053: writing smithi10548823-28 from 642067 to 1338255 tid 1 2024-04-20T02:51:24.765 INFO:tasks.rados.rados.0.smithi105.stdout:1053: writing smithi10548823-28 from 2025476 to 2484873 tid 2 2024-04-20T02:51:24.768 INFO:tasks.rados.rados.0.smithi105.stdout:1053: writing smithi10548823-28 from 2971091 to 3042067 tid 3 2024-04-20T02:51:24.768 INFO:tasks.rados.rados.0.smithi105.stdout:1054: write (excl) oid 39 current snap is 15 2024-04-20T02:51:24.769 INFO:tasks.rados.rados.0.smithi105.stdout:1054: seq_num 182 ranges {658874=771437,2083958=603055} 2024-04-20T02:51:24.776 INFO:tasks.rados.rados.0.smithi105.stdout:1054: writing smithi10548823-39 from 658874 to 1430311 tid 1 2024-04-20T02:51:24.785 INFO:tasks.rados.rados.0.smithi105.stdout:1054: writing smithi10548823-39 from 2083958 to 2687013 tid 2 2024-04-20T02:51:24.785 INFO:tasks.rados.rados.0.smithi105.stdout:1055: read oid 32 snap -1 2024-04-20T02:51:24.785 INFO:tasks.rados.rados.0.smithi105.stdout:1055: expect (ObjNum 174 snap 14 seq_num 174) 2024-04-20T02:51:24.785 INFO:tasks.rados.rados.0.smithi105.stdout:1056: snap_create 2024-04-20T02:51:24.794 INFO:tasks.rados.rados.0.smithi105.stdout:1055: expect (ObjNum 174 snap 14 seq_num 174) 2024-04-20T02:51:24.836 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:24.837+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:24.870 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:24.902 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:24.907 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:24.907 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:24.917 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:24.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:24.957 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:24.957 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:24.957+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:24.990 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:25.020 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:25.038 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:25.039+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:25.040 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:1053: finishing write tid 1 to smithi10548823-28 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:1053: finishing write tid 2 to smithi10548823-28 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:1053: finishing write tid 3 to smithi10548823-28 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:1053: finishing write tid 4 to smithi10548823-28 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:1053: finishing write tid 5 to smithi10548823-28 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:1053: oid 28 updating version 0 to 109 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:1053: oid 28 updating version 109 to 111 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:1053: oid 28 updating version 111 to 112 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:1053: oid 28 version 112 is already newer than 110 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 112 (ObjNum 181 snap 15 seq_num 181) dirty exists 2024-04-20T02:51:25.043 INFO:tasks.rados.rados.0.smithi105.stdout:1053: left oid 28 (ObjNum 181 snap 15 seq_num 181) 2024-04-20T02:51:25.044 INFO:tasks.rados.rados.0.smithi105.stdout:1054: finishing write tid 1 to smithi10548823-39 2024-04-20T02:51:25.044 INFO:tasks.rados.rados.0.smithi105.stdout:1054: finishing write tid 2 to smithi10548823-39 2024-04-20T02:51:25.044 INFO:tasks.rados.rados.0.smithi105.stdout:1054: finishing write tid 3 to smithi10548823-39 2024-04-20T02:51:25.044 INFO:tasks.rados.rados.0.smithi105.stdout:1054: finishing write tid 4 to smithi10548823-39 2024-04-20T02:51:25.044 INFO:tasks.rados.rados.0.smithi105.stdout:1054: oid 39 updating version 0 to 73 2024-04-20T02:51:25.044 INFO:tasks.rados.rados.0.smithi105.stdout:1054: oid 39 updating version 73 to 74 2024-04-20T02:51:25.044 INFO:tasks.rados.rados.0.smithi105.stdout:1054: oid 39 version 74 is already newer than 72 2024-04-20T02:51:25.044 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 74 (ObjNum 182 snap 15 seq_num 182) dirty exists 2024-04-20T02:51:25.044 INFO:tasks.rados.rados.0.smithi105.stdout:1054: left oid 39 (ObjNum 182 snap 15 seq_num 182) 2024-04-20T02:51:25.059 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:25.060 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:25.060 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:25.060 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:25.061 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:25.061 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:25.061 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:25.061 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:25.061 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:25.066 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:25.066 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:25.077 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:25.140 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:25.140+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:25.240 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:25.241+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:25.244 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:25.244 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:25.254 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:25.342 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:25.342+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:25.424 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:25.424 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:25.434 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:25.442 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:25.443+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:25.543 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:25.543+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:25.562 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:25.562 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:25.573 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:25.574 INFO:tasks.rados.rados.0.smithi105.stdout:1053: done (3 left) 2024-04-20T02:51:25.574 INFO:tasks.rados.rados.0.smithi105.stdout:1054: done (2 left) 2024-04-20T02:51:25.574 INFO:tasks.rados.rados.0.smithi105.stdout:1055: done (1 left) 2024-04-20T02:51:25.575 INFO:tasks.rados.rados.0.smithi105.stdout:1056: done (0 left) 2024-04-20T02:51:25.575 INFO:tasks.rados.rados.0.smithi105.stdout:1057: snap_create 2024-04-20T02:51:25.577 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:51:25.639 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:51:25.640 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:51:25.640 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-04-20T02:51:25.640 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-04-20T02:51:25.640 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:51:25.644 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:25.645+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:25.707 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:25.708 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:25.719 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:25.745 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:25.746+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:25.845 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:25.845+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:25.851 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:25.851 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:25.862 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:25.946 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:25.946+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:25.980 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:51:25.981 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:51:25.992 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-04-20T02:51:25.573336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":317381969,"num_objects":238,"num_object_clones":54,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":5,"num_read":1871,"num_read_kb":1527221,"num_write":1361,"num_write_kb":373058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":70676298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":61,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":856380,"kb_used_data":622400,"kb_used_omap":12,"kb_used_meta":233139,"kb_avail":754118340,"statfs":{"total":773094113280,"available":772217180160,"internally_reserved":0,"allocated":637337600,"data_stored":737306555,"data_compressed":14950972,"data_compressed_allocated":15593472,"data_compressed_original":36062761,"omap_allocated":12807,"internal_metadata":238734841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":69,"apply_latency_ms":69,"commit_latency_ns":69000000,"apply_latency_ns":69000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":54848482,"num_objects":31,"num_object_clones":23,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":35,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":1,"num_read":206,"num_read_kb":232422,"num_write":148,"num_write_kb":42254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":47229640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.079122"},"pg_stats":[{"pgid":"4.19","version":"64'157","reported_seq":368,"reported_epoch":65,"state":"peering","last_fresh":"2024-04-20T02:51:22.555362+0000","last_change":"2024-04-20T02:51:22.555362+0000","last_active":"2024-04-20T02:51:22.555179+0000","last_peered":"2024-04-20T02:51:21.971335+0000","last_clean":"2024-04-20T02:51:21.971335+0000","last_became_active":"2024-04-20T02:51:21.503563+0000","last_became_peered":"2024-04-20T02:51:21.503563+0000","last_unstale":"2024-04-20T02:51:22.555362+0000","last_undegraded":"2024-04-20T02:51:22.555362+0000","last_fullsized":"2024-04-20T02:51:22.555362+0000","mapping_epoch":65,"log_start":"59'150","ondisk_log_start":"59'150","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":150,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13796786,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":107,"num_read_kb":104733,"num_write":54,"num_write_kb":14624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7389695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.18","version":"64'49","reported_seq":175,"reported_epoch":65,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:22.555237+0000","last_change":"2024-04-20T02:51:21.539950+0000","last_active":"2024-04-20T02:51:22.555237+0000","last_peered":"2024-04-20T02:51:22.555237+0000","last_clean":"2024-04-20T02:50:58.916741+0000","last_became_active":"2024-04-20T02:51:21.510024+0000","last_became_peered":"2024-04-20T02:51:21.510024+0000","last_unstale":"2024-04-20T02:51:22.555237+0000","last_undegraded":"2024-04-20T02:51:22.555237+0000","last_fullsized":"2024-04-20T02:51:22.555237+0000","mapping_epoch":63,"log_start":"42'46","ondisk_log_start":"42'46","created":60,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":46,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5072097,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":65290,"num_write":20,"num_write_kb":4675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,7],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"61'47","reported_seq":222,"reported_epoch":66,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:23.556973+0000","last_change":"2024-04-20T02:51:21.564054+0000","last_active":"2024-04-20T02:51:23.556973+0000","last_peered":"2024-04-20T02:51:23.556973+0000","last_clean":"2024-04-20T02:51:18.558395+0000","last_became_active":"2024-04-20T02:51:21.512817+0000","last_became_peered":"2024-04-20T02:51:21.512817+0000","last_unstale":"2024-04-20T02:51:23.556973+0000","last_undegraded":"2024-04-20T02:51:23.556973+0000","last_fullsized":"2024-04-20T02:51:23.556973+0000","mapping_epoch":63,"log_start":"56'43","ondisk_log_start":"56'43","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":40,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6373913,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":59515,"num_write":23,"num_write_kb":6860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2757906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[0,1,5],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.1","version":"58'101","reported_seq":283,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-20T02:51:23.558682+0000","last_change":"2024-04-20T02:51:18.546413+0000","last_active":"2024-04-20T02:51:23.558682+0000","last_peered":"2024-04-20T02:51:23.558682+0000","last_clean":"2024-04-20T02:51:23.558682+0000","last_became_active":"2024-04-20T02:51:18.508043+0000","last_became_peered":"2024-04-20T02:51:18.508043+0000","last_unstale":"2024-04-20T02:51:23.558682+0000","last_undegraded":"2024-04-20T02:51:23.558682+0000","last_fullsized":"2024-04-20T02:51:23.558682+0000","mapping_epoch":46,"log_start":"57'97","ondisk_log_start":"57'97","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":97,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037943599000000001,"stat_sum":{"num_bytes":9042365,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":56,"num_read_kb":59027,"num_write":41,"num_write_kb":11120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3703652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.495939+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:51:21.495939+0000","last_peered":"2024-04-20T02:51:21.495939+0000","last_clean":"2024-04-20T02:51:21.495939+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:51:21.495939+0000","last_undegraded":"2024-04-20T02:51:21.495939+0000","last_fullsized":"2024-04-20T02:51:21.495939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.432280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"64'70","reported_seq":243,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.815372+0000","last_change":"2024-04-20T02:51:19.253156+0000","last_active":"2024-04-20T02:51:21.815372+0000","last_peered":"2024-04-20T02:51:21.815372+0000","last_clean":"2024-04-20T02:51:21.815372+0000","last_became_active":"2024-04-20T02:51:18.505708+0000","last_became_peered":"2024-04-20T02:51:18.505708+0000","last_unstale":"2024-04-20T02:51:21.815372+0000","last_undegraded":"2024-04-20T02:51:21.815372+0000","last_fullsized":"2024-04-20T02:51:21.815372+0000","mapping_epoch":22,"log_start":"53'65","ondisk_log_start":"53'65","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":63,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.922550+0000","scrub_duration":0.001820667,"objects_trimmed":0,"snaptrim_duration":0.020334135999999999,"stat_sum":{"num_bytes":7294205,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":56,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.496974+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:51:21.496974+0000","last_peered":"2024-04-20T02:51:21.496974+0000","last_clean":"2024-04-20T02:51:21.496974+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:51:21.496974+0000","last_undegraded":"2024-04-20T02:51:21.496974+0000","last_fullsized":"2024-04-20T02:51:21.496974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-20T02:51:22.558561+0000","last_change":"2024-04-20T02:50:28.091938+0000","last_active":"2024-04-20T02:51:22.558561+0000","last_peered":"2024-04-20T02:51:22.558561+0000","last_clean":"2024-04-20T02:51:22.558561+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:51:22.558561+0000","last_undegraded":"2024-04-20T02:51:22.558561+0000","last_fullsized":"2024-04-20T02:51:22.558561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:34.990718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"64'161","reported_seq":389,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-20T02:51:23.580666+0000","last_change":"2024-04-20T02:51:19.377353+0000","last_active":"2024-04-20T02:51:23.580666+0000","last_peered":"2024-04-20T02:51:23.580666+0000","last_clean":"2024-04-20T02:51:23.580666+0000","last_became_active":"2024-04-20T02:51:18.502004+0000","last_became_peered":"2024-04-20T02:51:18.502004+0000","last_unstale":"2024-04-20T02:51:23.580666+0000","last_undegraded":"2024-04-20T02:51:23.580666+0000","last_fullsized":"2024-04-20T02:51:23.580666+0000","mapping_epoch":47,"log_start":"61'157","ondisk_log_start":"61'157","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'157","last_scrub_stamp":"2024-04-20T02:51:19.377186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.377186+0000","objects_scrubbed":5,"log_size":4,"log_dups_size":153,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:38.408177+0000","scrub_duration":0.0026503899999999999,"objects_trimmed":0,"snaptrim_duration":0.018174327000000001,"stat_sum":{"num_bytes":8320028,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":20297,"num_write":4,"num_write_kb":1068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.1","version":"66'127","reported_seq":200,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-20T02:51:23.556695+0000","last_change":"2024-04-20T02:50:34.239216+0000","last_active":"2024-04-20T02:51:23.556695+0000","last_peered":"2024-04-20T02:51:23.556695+0000","last_clean":"2024-04-20T02:51:23.556695+0000","last_became_active":"2024-04-20T02:50:34.238906+0000","last_became_peered":"2024-04-20T02:50:34.238906+0000","last_unstale":"2024-04-20T02:51:23.556695+0000","last_undegraded":"2024-04-20T02:51:23.556695+0000","last_fullsized":"2024-04-20T02:51:23.556695+0000","mapping_epoch":21,"log_start":"59'124","ondisk_log_start":"59'124","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":149,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.390119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33880165,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":118,"num_read_kb":6387,"num_write":154,"num_write_kb":53016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":56,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-20T02:51:22.558065+0000","last_change":"2024-04-20T02:50:28.092115+0000","last_active":"2024-04-20T02:51:22.558065+0000","last_peered":"2024-04-20T02:51:22.558065+0000","last_clean":"2024-04-20T02:51:22.558065+0000","last_became_active":"2024-04-20T02:50:28.091570+0000","last_became_peered":"2024-04-20T02:50:28.091570+0000","last_unstale":"2024-04-20T02:51:22.558065+0000","last_undegraded":"2024-04-20T02:51:22.558065+0000","last_fullsized":"2024-04-20T02:51:22.558065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:27.403210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'61","reported_seq":220,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-20T02:51:23.570655+0000","last_change":"2024-04-20T02:51:21.386970+0000","last_active":"2024-04-20T02:51:23.570655+0000","last_peered":"2024-04-20T02:51:23.570655+0000","last_clean":"2024-04-20T02:51:23.570655+0000","last_became_active":"2024-04-20T02:51:18.504889+0000","last_became_peered":"2024-04-20T02:51:18.504889+0000","last_unstale":"2024-04-20T02:51:23.570655+0000","last_undegraded":"2024-04-20T02:51:23.570655+0000","last_fullsized":"2024-04-20T02:51:23.570655+0000","mapping_epoch":22,"log_start":"59'57","ondisk_log_start":"59'57","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'57","last_scrub_stamp":"2024-04-20T02:51:21.386796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:21.386796+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":56,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:27.553181+0000","scrub_duration":0.0018878359999999999,"objects_trimmed":0,"snaptrim_duration":0.002783518,"stat_sum":{"num_bytes":6446814,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":11428,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.2","version":"53'105","reported_seq":258,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.512647+0000","last_change":"2024-04-20T02:51:07.118974+0000","last_active":"2024-04-20T02:51:21.512647+0000","last_peered":"2024-04-20T02:51:21.512647+0000","last_clean":"2024-04-20T02:51:21.512647+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:51:21.512647+0000","last_undegraded":"2024-04-20T02:51:21.512647+0000","last_fullsized":"2024-04-20T02:51:21.512647+0000","mapping_epoch":49,"log_start":"52'102","ondisk_log_start":"52'102","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":110,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:42.672805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27256345,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":95,"num_read_kb":7581,"num_write":122,"num_write_kb":50638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":209,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-20T02:51:22.558021+0000","last_change":"2024-04-20T02:50:19.923431+0000","last_active":"2024-04-20T02:51:22.558021+0000","last_peered":"2024-04-20T02:51:22.558021+0000","last_clean":"2024-04-20T02:51:22.558021+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:51:22.558021+0000","last_undegraded":"2024-04-20T02:51:22.558021+0000","last_fullsized":"2024-04-20T02:51:22.558021+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:50:17.569030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.655607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":32,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.504217+0000","last_change":"2024-04-20T02:51:01.151894+0000","last_active":"2024-04-20T02:51:21.504217+0000","last_peered":"2024-04-20T02:51:21.504217+0000","last_clean":"2024-04-20T02:51:21.504217+0000","last_became_active":"2024-04-20T02:51:01.151544+0000","last_became_peered":"2024-04-20T02:51:01.151544+0000","last_unstale":"2024-04-20T02:51:21.504217+0000","last_undegraded":"2024-04-20T02:51:21.504217+0000","last_fullsized":"2024-04-20T02:51:21.504217+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:29.245779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"56'33","reported_seq":159,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.496204+0000","last_change":"2024-04-20T02:51:18.503259+0000","last_active":"2024-04-20T02:51:21.496204+0000","last_peered":"2024-04-20T02:51:21.496204+0000","last_clean":"2024-04-20T02:51:21.496204+0000","last_became_active":"2024-04-20T02:51:18.502179+0000","last_became_peered":"2024-04-20T02:51:18.502179+0000","last_unstale":"2024-04-20T02:51:21.496204+0000","last_undegraded":"2024-04-20T02:51:21.496204+0000","last_fullsized":"2024-04-20T02:51:21.496204+0000","mapping_epoch":22,"log_start":"53'30","ondisk_log_start":"53'30","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060446299999999996,"stat_sum":{"num_bytes":4739995,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":44474,"num_write":11,"num_write_kb":3858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.0","version":"59'97","reported_seq":158,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:22.162211+0000","last_change":"2024-04-20T02:50:34.241909+0000","last_active":"2024-04-20T02:51:22.162211+0000","last_peered":"2024-04-20T02:51:22.162211+0000","last_clean":"2024-04-20T02:51:22.162211+0000","last_became_active":"2024-04-20T02:50:34.241613+0000","last_became_peered":"2024-04-20T02:50:34.241613+0000","last_unstale":"2024-04-20T02:51:22.162211+0000","last_undegraded":"2024-04-20T02:51:22.162211+0000","last_fullsized":"2024-04-20T02:51:22.162211+0000","mapping_epoch":21,"log_start":"55'94","ondisk_log_start":"55'94","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":106,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:54.710965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23879111,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":84,"num_read_kb":3291,"num_write":126,"num_write_kb":40638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.501974+0000","last_change":"2024-04-20T02:50:28.091742+0000","last_active":"2024-04-20T02:51:21.501974+0000","last_peered":"2024-04-20T02:51:21.501974+0000","last_clean":"2024-04-20T02:51:21.501974+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:51:21.501974+0000","last_undegraded":"2024-04-20T02:51:21.501974+0000","last_fullsized":"2024-04-20T02:51:21.501974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:32.326186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"56'43","reported_seq":210,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.496534+0000","last_change":"2024-04-20T02:51:18.507004+0000","last_active":"2024-04-20T02:51:21.496534+0000","last_peered":"2024-04-20T02:51:21.496534+0000","last_clean":"2024-04-20T02:51:21.496534+0000","last_became_active":"2024-04-20T02:51:18.506025+0000","last_became_peered":"2024-04-20T02:51:18.506025+0000","last_unstale":"2024-04-20T02:51:21.496534+0000","last_undegraded":"2024-04-20T02:51:21.496534+0000","last_fullsized":"2024-04-20T02:51:21.496534+0000","mapping_epoch":46,"log_start":"55'40","ondisk_log_start":"55'40","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050649899999999999,"stat_sum":{"num_bytes":5080805,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":42186,"num_write":19,"num_write_kb":5596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2757905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.3","version":"64'96","reported_seq":161,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-20T02:51:22.555429+0000","last_change":"2024-04-20T02:50:34.242061+0000","last_active":"2024-04-20T02:51:22.555429+0000","last_peered":"2024-04-20T02:51:22.555429+0000","last_clean":"2024-04-20T02:51:22.555429+0000","last_became_active":"2024-04-20T02:50:34.241638+0000","last_became_peered":"2024-04-20T02:50:34.241638+0000","last_unstale":"2024-04-20T02:51:22.555429+0000","last_undegraded":"2024-04-20T02:51:22.555429+0000","last_fullsized":"2024-04-20T02:51:22.555429+0000","mapping_epoch":21,"log_start":"54'93","ondisk_log_start":"54'93","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":99,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:58.050023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29502528,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":87,"num_read_kb":3888,"num_write":121,"num_write_kb":39133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":32,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.504098+0000","last_change":"2024-04-20T02:51:01.134074+0000","last_active":"2024-04-20T02:51:21.504098+0000","last_peered":"2024-04-20T02:51:21.504098+0000","last_clean":"2024-04-20T02:51:21.504098+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:51:21.504098+0000","last_undegraded":"2024-04-20T02:51:21.504098+0000","last_fullsized":"2024-04-20T02:51:21.504098+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:38.091564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"56'17","reported_seq":145,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:22.420505+0000","last_change":"2024-04-20T02:51:22.420505+0000","last_active":"2024-04-20T02:51:22.420505+0000","last_peered":"2024-04-20T02:51:22.420505+0000","last_clean":"2024-04-20T02:51:22.420505+0000","last_became_active":"2024-04-20T02:51:18.509276+0000","last_became_peered":"2024-04-20T02:51:18.509276+0000","last_unstale":"2024-04-20T02:51:22.420505+0000","last_undegraded":"2024-04-20T02:51:22.420505+0000","last_fullsized":"2024-04-20T02:51:22.420505+0000","mapping_epoch":22,"log_start":"53'14","ondisk_log_start":"53'14","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:51:22.420320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:22.420320+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.257627+0000","scrub_duration":0.0016858400000000001,"objects_trimmed":0,"snaptrim_duration":0.0021724610000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.2","version":"0'0","reported_seq":118,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.496004+0000","last_change":"2024-04-20T02:51:18.507329+0000","last_active":"2024-04-20T02:51:21.496004+0000","last_peered":"2024-04-20T02:51:21.496004+0000","last_clean":"2024-04-20T02:51:21.496004+0000","last_became_active":"2024-04-20T02:51:18.506462+0000","last_became_peered":"2024-04-20T02:51:18.506462+0000","last_unstale":"2024-04-20T02:51:21.496004+0000","last_undegraded":"2024-04-20T02:51:21.496004+0000","last_fullsized":"2024-04-20T02:51:21.496004+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052908199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":56,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.496909+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:51:21.496909+0000","last_peered":"2024-04-20T02:51:21.496909+0000","last_clean":"2024-04-20T02:51:21.496909+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:51:21.496909+0000","last_undegraded":"2024-04-20T02:51:21.496909+0000","last_fullsized":"2024-04-20T02:51:21.496909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"48'47","reported_seq":175,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-20T02:51:22.554565+0000","last_change":"2024-04-20T02:51:20.501867+0000","last_active":"2024-04-20T02:51:22.554565+0000","last_peered":"2024-04-20T02:51:22.554565+0000","last_clean":"2024-04-20T02:51:22.554565+0000","last_became_active":"2024-04-20T02:51:20.501370+0000","last_became_peered":"2024-04-20T02:51:20.501370+0000","last_unstale":"2024-04-20T02:51:22.554565+0000","last_undegraded":"2024-04-20T02:51:22.554565+0000","last_fullsized":"2024-04-20T02:51:22.554565+0000","mapping_epoch":62,"log_start":"42'44","ondisk_log_start":"42'44","created":22,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":44,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.044654740999999998,"stat_sum":{"num_bytes":5072097,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":41631,"num_write":18,"num_write_kb":4274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.9","version":"66'162","reported_seq":336,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-20T02:51:23.605428+0000","last_change":"2024-04-20T02:51:18.549053+0000","last_active":"2024-04-20T02:51:23.605428+0000","last_peered":"2024-04-20T02:51:23.605428+0000","last_clean":"2024-04-20T02:51:23.605428+0000","last_became_active":"2024-04-20T02:51:18.509871+0000","last_became_peered":"2024-04-20T02:51:18.509871+0000","last_unstale":"2024-04-20T02:51:23.605428+0000","last_undegraded":"2024-04-20T02:51:23.605428+0000","last_fullsized":"2024-04-20T02:51:23.605428+0000","mapping_epoch":22,"log_start":"65'157","ondisk_log_start":"65'157","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":157,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.037969359000000001,"stat_sum":{"num_bytes":15535224,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":110,"num_read_kb":110552,"num_write":61,"num_write_kb":16725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.a","version":"53'33","reported_seq":164,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.495757+0000","last_change":"2024-04-20T02:51:17.489866+0000","last_active":"2024-04-20T02:51:21.495757+0000","last_peered":"2024-04-20T02:51:21.495757+0000","last_clean":"2024-04-20T02:51:21.495757+0000","last_became_active":"2024-04-20T02:50:35.413503+0000","last_became_peered":"2024-04-20T02:50:35.413503+0000","last_unstale":"2024-04-20T02:51:21.495757+0000","last_undegraded":"2024-04-20T02:51:21.495757+0000","last_fullsized":"2024-04-20T02:51:21.495757+0000","mapping_epoch":22,"log_start":"44'29","ondisk_log_start":"44'29","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":29,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:57.185462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040938999999999998,"stat_sum":{"num_bytes":8235364,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":108,"num_read_kb":124480,"num_write":28,"num_write_kb":7354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.b","version":"64'85","reported_seq":223,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-20T02:51:24.173204+0000","last_change":"2024-04-20T02:51:17.499992+0000","last_active":"2024-04-20T02:51:24.173204+0000","last_peered":"2024-04-20T02:51:24.173204+0000","last_clean":"2024-04-20T02:51:24.173204+0000","last_became_active":"2024-04-20T02:50:35.242331+0000","last_became_peered":"2024-04-20T02:50:35.242331+0000","last_unstale":"2024-04-20T02:51:24.173204+0000","last_undegraded":"2024-04-20T02:51:24.173204+0000","last_fullsized":"2024-04-20T02:51:24.173204+0000","mapping_epoch":22,"log_start":"64'80","ondisk_log_start":"64'80","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":80,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.773051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069626410000000003,"stat_sum":{"num_bytes":16327627,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":117,"num_read_kb":137866,"num_write":66,"num_write_kb":17869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.c","version":"64'53","reported_seq":165,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-20T02:51:22.555528+0000","last_change":"2024-04-20T02:51:17.499433+0000","last_active":"2024-04-20T02:51:22.555528+0000","last_peered":"2024-04-20T02:51:22.555528+0000","last_clean":"2024-04-20T02:51:22.555528+0000","last_became_active":"2024-04-20T02:50:35.241480+0000","last_became_peered":"2024-04-20T02:50:35.241480+0000","last_unstale":"2024-04-20T02:51:22.555528+0000","last_undegraded":"2024-04-20T02:51:22.555528+0000","last_fullsized":"2024-04-20T02:51:22.555528+0000","mapping_epoch":22,"log_start":"64'49","ondisk_log_start":"64'49","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":49,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.915698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024414480000000001,"stat_sum":{"num_bytes":10579498,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":50,"num_read_kb":49915,"num_write":40,"num_write_kb":11356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.d","version":"64'92","reported_seq":249,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-20T02:51:23.558365+0000","last_change":"2024-04-20T02:51:17.500309+0000","last_active":"2024-04-20T02:51:23.558365+0000","last_peered":"2024-04-20T02:51:23.558365+0000","last_clean":"2024-04-20T02:51:23.558365+0000","last_became_active":"2024-04-20T02:50:35.528107+0000","last_became_peered":"2024-04-20T02:50:35.528107+0000","last_unstale":"2024-04-20T02:51:23.558365+0000","last_undegraded":"2024-04-20T02:51:23.558365+0000","last_fullsized":"2024-04-20T02:51:23.558365+0000","mapping_epoch":22,"log_start":"56'88","ondisk_log_start":"56'88","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":88,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:59.747989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023033039999999999,"stat_sum":{"num_bytes":20497267,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":188,"num_read_kb":207534,"num_write":63,"num_write_kb":17969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.e","version":"64'41","reported_seq":149,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-20T02:51:22.555538+0000","last_change":"2024-04-20T02:51:17.499684+0000","last_active":"2024-04-20T02:51:22.555538+0000","last_peered":"2024-04-20T02:51:22.555538+0000","last_clean":"2024-04-20T02:51:22.555538+0000","last_became_active":"2024-04-20T02:50:35.244872+0000","last_became_peered":"2024-04-20T02:50:35.244872+0000","last_unstale":"2024-04-20T02:51:22.555538+0000","last_undegraded":"2024-04-20T02:51:22.555538+0000","last_fullsized":"2024-04-20T02:51:22.555538+0000","mapping_epoch":22,"log_start":"55'38","ondisk_log_start":"55'38","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:39.148211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032048129999999999,"stat_sum":{"num_bytes":7252465,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":36170,"num_write":25,"num_write_kb":9031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.f","version":"34'15","reported_seq":113,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.496629+0000","last_change":"2024-04-20T02:51:17.499778+0000","last_active":"2024-04-20T02:51:21.496629+0000","last_peered":"2024-04-20T02:51:21.496629+0000","last_clean":"2024-04-20T02:51:21.496629+0000","last_became_active":"2024-04-20T02:50:35.245323+0000","last_became_peered":"2024-04-20T02:50:35.245323+0000","last_unstale":"2024-04-20T02:51:21.496629+0000","last_undegraded":"2024-04-20T02:51:21.496629+0000","last_fullsized":"2024-04-20T02:51:21.496629+0000","mapping_epoch":22,"log_start":"33'10","ondisk_log_start":"33'10","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":10,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:53.742736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000442161,"stat_sum":{"num_bytes":3428595,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":8320,"num_write":14,"num_write_kb":4030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.10","version":"64'71","reported_seq":250,"reported_epoch":64,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:21.814806+0000","last_change":"2024-04-20T02:51:21.542805+0000","last_active":"2024-04-20T02:51:21.814806+0000","last_peered":"2024-04-20T02:51:21.814806+0000","last_clean":"2024-04-20T02:51:18.527159+0000","last_became_active":"2024-04-20T02:51:21.508521+0000","last_became_peered":"2024-04-20T02:51:21.508521+0000","last_unstale":"2024-04-20T02:51:21.814806+0000","last_undegraded":"2024-04-20T02:51:21.814806+0000","last_fullsized":"2024-04-20T02:51:21.814806+0000","mapping_epoch":63,"log_start":"54'67","ondisk_log_start":"54'67","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":65,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017870196000000001,"stat_sum":{"num_bytes":6729079,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":88820,"num_write":26,"num_write_kb":6965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,0,7],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.11","version":"61'103","reported_seq":313,"reported_epoch":65,"state":"peering","last_fresh":"2024-04-20T02:51:22.546961+0000","last_change":"2024-04-20T02:51:22.588101+0000","last_active":"2024-04-20T02:51:22.588101+0000","last_peered":"2024-04-20T02:51:22.546961+0000","last_clean":"2024-04-20T02:51:22.546961+0000","last_became_active":"2024-04-20T02:51:21.510115+0000","last_became_peered":"2024-04-20T02:51:21.510115+0000","last_unstale":"2024-04-20T02:51:22.546961+0000","last_undegraded":"2024-04-20T02:51:22.546961+0000","last_fullsized":"2024-04-20T02:51:22.546961+0000","mapping_epoch":65,"log_start":"58'101","ondisk_log_start":"58'101","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":101,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.043425455000000002,"stat_sum":{"num_bytes":9042366,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":48,"num_read_kb":43869,"num_write":41,"num_write_kb":11120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":15332310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.12","version":"0'0","reported_seq":127,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-20T02:51:21.495794+0000","last_change":"2024-04-20T02:51:20.505392+0000","last_active":"2024-04-20T02:51:21.495794+0000","last_peered":"2024-04-20T02:51:21.495794+0000","last_clean":"2024-04-20T02:51:21.495794+0000","last_became_active":"2024-04-20T02:51:20.504935+0000","last_became_peered":"2024-04-20T02:51:20.504935+0000","last_unstale":"2024-04-20T02:51:21.495794+0000","last_undegraded":"2024-04-20T02:51:21.495794+0000","last_fullsized":"2024-04-20T02:51:21.495794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075727599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.13","version":"65'161","reported_seq":388,"reported_epoch":66,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:23.558371+0000","last_change":"2024-04-20T02:51:21.522920+0000","last_active":"2024-04-20T02:51:23.558371+0000","last_peered":"2024-04-20T02:51:23.558371+0000","last_clean":"2024-04-20T02:51:18.548102+0000","last_became_active":"2024-04-20T02:51:21.509184+0000","last_became_peered":"2024-04-20T02:51:21.509184+0000","last_unstale":"2024-04-20T02:51:23.558371+0000","last_undegraded":"2024-04-20T02:51:23.558371+0000","last_fullsized":"2024-04-20T02:51:23.558371+0000","mapping_epoch":63,"log_start":"64'157","ondisk_log_start":"64'157","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":157,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072187459999999998,"stat_sum":{"num_bytes":16772086,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":94,"num_read_kb":106676,"num_write":59,"num_write_kb":15381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9140440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.14","version":"64'22","reported_seq":129,"reported_epoch":65,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:22.554018+0000","last_change":"2024-04-20T02:51:21.541565+0000","last_active":"2024-04-20T02:51:22.554018+0000","last_peered":"2024-04-20T02:51:22.554018+0000","last_clean":"2024-04-20T02:51:14.032218+0000","last_became_active":"2024-04-20T02:51:21.510278+0000","last_became_peered":"2024-04-20T02:51:21.510278+0000","last_unstale":"2024-04-20T02:51:22.554018+0000","last_undegraded":"2024-04-20T02:51:22.554018+0000","last_fullsized":"2024-04-20T02:51:22.554018+0000","mapping_epoch":63,"log_start":"56'17","ondisk_log_start":"56'17","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":17,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3093185,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":26723,"num_write":10,"num_write_kb":2705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,2],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.15","version":"64'38","reported_seq":151,"reported_epoch":65,"state":"peering","last_fresh":"2024-04-20T02:51:22.555018+0000","last_change":"2024-04-20T02:51:22.555018+0000","last_active":"2024-04-20T02:51:22.555018+0000","last_peered":"2024-04-20T02:51:21.909546+0000","last_clean":"2024-04-20T02:51:21.909546+0000","last_became_active":"2024-04-20T02:51:21.508321+0000","last_became_peered":"2024-04-20T02:51:21.508321+0000","last_unstale":"2024-04-20T02:51:22.555018+0000","last_undegraded":"2024-04-20T02:51:22.555018+0000","last_fullsized":"2024-04-20T02:51:22.555018+0000","mapping_epoch":65,"log_start":"56'33","ondisk_log_start":"56'33","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":33,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5926755,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":44474,"num_write":17,"num_write_kb":5019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6513594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.16","version":"64'61","reported_seq":247,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-04-20T02:51:23.562336+0000","last_change":"2024-04-20T02:51:23.562336+0000","last_active":"2024-04-20T02:51:23.562336+0000","last_peered":"2024-04-20T02:51:23.562336+0000","last_clean":"2024-04-20T02:51:23.562336+0000","last_became_active":"2024-04-20T02:51:23.561843+0000","last_became_peered":"2024-04-20T02:51:23.561843+0000","last_unstale":"2024-04-20T02:51:23.562336+0000","last_undegraded":"2024-04-20T02:51:23.562336+0000","last_fullsized":"2024-04-20T02:51:23.562336+0000","mapping_epoch":65,"log_start":"59'57","ondisk_log_start":"59'57","created":60,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":55,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083454640000000004,"stat_sum":{"num_bytes":7614836,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":64,"num_read_kb":71993,"num_write":33,"num_write_kb":8547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8853601,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":202273452,"num_objects":100,"num_object_clones":54,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":5,"num_read":1371,"num_read_kb":1505974,"num_write":679,"num_write_kb":187217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":56449103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":61,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":449085440,"data_stored":536841774,"data_compressed":2462704,"data_compressed_allocated":2207744,"data_compressed_original":5536514,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":114518149,"num_objects":136,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":384,"num_read_kb":21147,"num_write":523,"num_write_kb":183425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186228736,"data_stored":201382890,"data_compressed":12526532,"data_compressed_allocated":13381632,"data_compressed_original":29126656,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509459,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96732,"kb_used_data":65040,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94275108,"statfs":{"total":96636764160,"available":96537710592,"internally_reserved":0,"allocated":66600960,"data_stored":78677535,"data_compressed":259057,"data_compressed_allocated":294912,"data_compressed_original":1011712,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.376},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":1.8069999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":8.9369999999999994},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":9.8119999999999994}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.314},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":9.6440000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.77300000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.33000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.34499999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757255,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89400,"kb_used_data":61676,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94282440,"statfs":{"total":96636764160,"available":96545218560,"internally_reserved":0,"allocated":63156224,"data_stored":73858152,"data_compressed":1083307,"data_compressed_allocated":1159168,"data_compressed_original":2473984,"omap_allocated":1657,"internal_metadata":28375431},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74344,"kb_used_data":47320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94297496,"statfs":{"total":96636764160,"available":96560635904,"internally_reserved":0,"allocated":48455680,"data_stored":56871513,"data_compressed":1261942,"data_compressed_allocated":1232896,"data_compressed_original":2727936,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.28000000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.245}]},{"osd":3,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":11.877000000000001,"5min":11.877000000000001,"15min":11.877000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":6,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":11.805,"5min":11.805,"15min":11.805},"last":0.441}]},{"osd":7,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.40200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111392,"kb_used_data":84368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94260448,"statfs":{"total":96636764160,"available":96522698752,"internally_reserved":0,"allocated":86392832,"data_stored":102077621,"data_compressed":1625498,"data_compressed_allocated":1593344,"data_compressed_original":3694166,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.34100000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.312}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.80500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121580,"kb_used_data":94564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94250260,"statfs":{"total":96636764160,"available":96512266240,"internally_reserved":0,"allocated":96833536,"data_stored":108870207,"data_compressed":2455258,"data_compressed_allocated":2592768,"data_compressed_original":5557846,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.253},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.314}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.379},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.501},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.48799999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.59699999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112568,"kb_used_data":81300,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94259272,"statfs":{"total":96636764160,"available":96521494528,"internally_reserved":0,"allocated":83251200,"data_stored":92291232,"data_compressed":3243692,"data_compressed_allocated":3440640,"data_compressed_original":8051495,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.0369999999999999,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.53800000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.73099999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":9.9570000000000007,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":128008,"kb_used_data":96892,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94243832,"statfs":{"total":96636764160,"available":96505683968,"internally_reserved":0,"allocated":99217408,"data_stored":117381390,"data_compressed":2518021,"data_compressed_allocated":2625536,"data_compressed_original":6504448,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":1.427},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.81799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":1.4990000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.93200000000000005}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":3.2730000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":1.728}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.5349999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":1.4690000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.878},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.69299999999999995}]}]},{"osd":4,"up_from":15,"seq":64424509457,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122356,"kb_used_data":91240,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94249484,"statfs":{"total":96636764160,"available":96511471616,"internally_reserved":0,"allocated":93429760,"data_stored":107278905,"data_compressed":2504197,"data_compressed_allocated":2654208,"data_compressed_original":6041174,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.39500000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.378},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.434}]},{"osd":2,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.40600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.35399999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.309},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.46000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34942976,"data_stored":37725185,"data_compressed":2169376,"data_compressed_allocated":2326528,"data_compressed_original":5206016,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13307904,"data_stored":14412936,"data_compressed":977598,"data_compressed_allocated":1036288,"data_compressed_original":2203648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13307904,"data_stored":14412936,"data_compressed":977598,"data_compressed_allocated":1036288,"data_compressed_original":2203648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31117312,"data_stored":33550824,"data_compressed":2087841,"data_compressed_allocated":2228224,"data_compressed_original":4747264,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34942976,"data_stored":37725185,"data_compressed":2169376,"data_compressed_allocated":2326528,"data_compressed_original":5206016,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14135296,"data_stored":15322309,"data_compressed":1048539,"data_compressed_allocated":1118208,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":44474368,"data_stored":48233515,"data_compressed":3096204,"data_compressed_allocated":3309568,"data_compressed_original":7217152,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":63483904,"data_stored":78816656,"data_compressed":343327,"data_compressed_allocated":266240,"data_compressed_original":786432,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":72372224,"data_stored":87365247,"data_compressed":647900,"data_compressed_allocated":557056,"data_compressed_original":1490518,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34435072,"data_stored":42159139,"data_compressed":284344,"data_compressed_allocated":196608,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65003520,"data_stored":75019945,"data_compressed":367417,"data_compressed_allocated":364544,"data_compressed_original":810582,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58023936,"data_stored":69570779,"data_compressed":346329,"data_compressed_allocated":339968,"data_compressed_original":900694,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":48295936,"data_stored":58230951,"data_compressed":34768,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":41660416,"data_stored":47841071,"data_compressed":184880,"data_compressed_allocated":180224,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65810432,"data_stored":77837986,"data_compressed":253739,"data_compressed_allocated":262144,"data_compressed_original":499712,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:51:25.996 INFO:tasks.ceph.ceph_manager.ceph:decrease pool size by 10 2024-04-20T02:51:25.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 16 2024-04-20T02:51:26.004 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:26.005 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:26.015 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:26.059 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:26.060+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:26.117 INFO:tasks.rados.rados.0.smithi105.stdout:1057: done (0 left) 2024-04-20T02:51:26.117 INFO:tasks.rados.rados.0.smithi105.stdout:1058: tier_evict oid 13 2024-04-20T02:51:26.117 INFO:tasks.rados.rados.0.smithi105.stdout:1058: tier_evict oid 13 snap -1 2024-04-20T02:51:26.117 INFO:tasks.rados.rados.0.smithi105.stdout:1059: read oid 42 snap 6 2024-04-20T02:51:26.117 INFO:tasks.rados.rados.0.smithi105.stdout:1059: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:51:26.117 INFO:tasks.rados.rados.0.smithi105.stdout:1060: read oid 5 snap -1 2024-04-20T02:51:26.118 INFO:tasks.rados.rados.0.smithi105.stdout:1060: expect deleted 2024-04-20T02:51:26.118 INFO:tasks.rados.rados.0.smithi105.stdout:1061: write oid 32 current snap is 17 2024-04-20T02:51:26.118 INFO:tasks.rados.rados.0.smithi105.stdout:1061: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-04-20T02:51:26.124 INFO:tasks.rados.rados.0.smithi105.stdout:1061: writing smithi10548823-32 from 675681 to 1238720 tid 1 2024-04-20T02:51:26.132 INFO:tasks.rados.rados.0.smithi105.stdout:1061: writing smithi10548823-32 from 1858793 to 2605506 tid 2 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1061: writing smithi10548823-32 from 3025937 to 3075681 tid 3 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1062: read oid 49 snap -1 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1062: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1063: read oid 1 snap -1 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1063: expect (ObjNum 133 snap 9 seq_num 133) 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1064: tier_evict oid 20 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1064: tier_evict oid 20 snap -1 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1060: done (6 left) 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1065: read oid 48 snap -1 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1065: expect (ObjNum 154 snap 10 seq_num 154) 2024-04-20T02:51:26.136 INFO:tasks.rados.rados.0.smithi105.stdout:1066: write (excl) oid 37 current snap is 17 2024-04-20T02:51:26.137 INFO:tasks.rados.rados.0.smithi105.stdout:1058: got (22) Invalid argument 2024-04-20T02:51:26.137 INFO:tasks.rados.rados.0.smithi105.stdout:1066: seq_num 184 ranges {692488=638288,1917275=606724} 2024-04-20T02:51:26.143 INFO:tasks.rados.rados.0.smithi105.stdout:1066: writing smithi10548823-37 from 692488 to 1330776 tid 1 2024-04-20T02:51:26.148 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:26.149+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:26.151 INFO:tasks.rados.rados.0.smithi105.stdout:1066: writing smithi10548823-37 from 1917275 to 2523999 tid 2 2024-04-20T02:51:26.151 INFO:tasks.rados.rados.0.smithi105.stdout:1058: done (7 left) 2024-04-20T02:51:26.151 INFO:tasks.rados.rados.0.smithi105.stdout:1067: read oid 2 snap -1 2024-04-20T02:51:26.151 INFO:tasks.rados.rados.0.smithi105.stdout:1067: expect (ObjNum 155 snap 10 seq_num 155) 2024-04-20T02:51:26.151 INFO:tasks.rados.rados.0.smithi105.stdout:1068: read oid 12 snap -1 2024-04-20T02:51:26.151 INFO:tasks.rados.rados.0.smithi105.stdout:1068: expect (ObjNum 167 snap 14 seq_num 167) 2024-04-20T02:51:26.152 INFO:tasks.rados.rados.0.smithi105.stdout:1069: set_chunk oid 4 target oid 2024-04-20T02:51:26.162 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:26.162 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:26.177 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:26.184 INFO:tasks.rados.rados.0.smithi105.stdout:1069: set_chunk oid 4 offset: 1421824 length: 1240064 target oid 2b4532a5e40cb89ac9b2ab8f771403e620e0dcfb57f4437686f34c28f23d41da offset: 0 snap 6 2024-04-20T02:51:26.184 INFO:tasks.rados.rados.0.smithi105.stdout:1070: read oid 6 snap -1 2024-04-20T02:51:26.184 INFO:tasks.rados.rados.0.smithi105.stdout:1070: expect deleted 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1071: read oid 28 snap -1 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1071: expect (ObjNum 181 snap 15 seq_num 181) 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1072: tier_promote oid 3 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1073: read oid 41 snap -1 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1073: expect (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1074: tier_evict oid 22 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1074: tier_evict oid 22 snap -1 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1075: read oid 8 snap -1 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1075: expect (ObjNum 86 snap 2 seq_num 86) 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1064: got (22) Invalid argument 2024-04-20T02:51:26.185 INFO:tasks.rados.rados.0.smithi105.stdout:1065: expect (ObjNum 154 snap 10 seq_num 154) 2024-04-20T02:51:26.260 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:26.261+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:26.344 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:26.344 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:26.350 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:26.351+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:26.354 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:26.363 INFO:tasks.rados.rados.0.smithi105.stdout:1063: expect (ObjNum 133 snap 9 seq_num 133) 2024-04-20T02:51:26.451 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:26.452+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:26.490 INFO:tasks.rados.rados.0.smithi105.stdout:1059: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:51:26.524 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:26.524 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:26.533 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:26.552 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:26.552+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:26.654 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:26.653+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:26.664 INFO:tasks.rados.rados.0.smithi105.stdout:1062: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:26.701 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:26.702 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:26.713 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:26.758 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:26.759+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:26.802 INFO:tasks.rados.rados.0.smithi105.stdout:1068: expect (ObjNum 167 snap 14 seq_num 167) 2024-04-20T02:51:26.842 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:26.842 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:26.851 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:26.855 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:26.855+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:26.945 INFO:tasks.rados.rados.0.smithi105.stdout:1067: expect (ObjNum 155 snap 10 seq_num 155) 2024-04-20T02:51:26.956 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:26.956+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:27.021 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:27.022 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:27.034 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:27.056 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:27.057+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:27.073 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1061: finishing write tid 1 to smithi10548823-32 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1061: finishing write tid 2 to smithi10548823-32 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1059: done (15 left) 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1062: done (14 left) 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1063: done (13 left) 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1064: done (12 left) 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1065: done (11 left) 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1067: done (10 left) 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1068: done (9 left) 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1076: write (excl) oid 30 current snap is 17 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1061: finishing write tid 3 to smithi10548823-32 2024-04-20T02:51:27.076 INFO:tasks.rados.rados.0.smithi105.stdout:1076: seq_num 185 ranges {709295=713537,1859404=750382} 2024-04-20T02:51:27.083 INFO:tasks.rados.rados.0.smithi105.stdout:1076: writing smithi10548823-30 from 709295 to 1422832 tid 1 2024-04-20T02:51:27.090 INFO:tasks.rados.rados.0.smithi105.stdout:1076: writing smithi10548823-30 from 1859404 to 2609786 tid 2 2024-04-20T02:51:27.090 INFO:tasks.rados.rados.0.smithi105.stdout:1077: read oid 19 snap -1 2024-04-20T02:51:27.090 INFO:tasks.rados.rados.0.smithi105.stdout:1077: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:27.090 INFO:tasks.rados.rados.0.smithi105.stdout:1078: tier_evict oid 43 2024-04-20T02:51:27.090 INFO:tasks.rados.rados.0.smithi105.stdout:1078: tier_evict oid 43 snap -1 2024-04-20T02:51:27.090 INFO:tasks.rados.rados.0.smithi105.stdout:1079: set_chunk oid 35 target oid 2024-04-20T02:51:27.099 INFO:tasks.rados.rados.0.smithi105.stdout:1079: set_chunk oid 35 offset: 1409536 length: 595968 target oid 24779ae201e9e2330f14fb54d4c17fff4fcc44403303f238b3af4a57b584d69f offset: 0 snap 13 2024-04-20T02:51:27.099 INFO:tasks.rados.rados.0.smithi105.stdout:1080: snap_create 2024-04-20T02:51:27.099 INFO:tasks.rados.rados.0.smithi105.stdout:1061: finishing write tid 4 to smithi10548823-32 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:1061: finishing write tid 5 to smithi10548823-32 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:1061: oid 32 updating version 0 to 42 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:1061: oid 32 version 42 is already newer than 41 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:1061: oid 32 version 42 is already newer than 40 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:1061: oid 32 updating version 42 to 43 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 43 (ObjNum 183 snap 17 seq_num 183) dirty exists 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:1061: left oid 32 (ObjNum 183 snap 17 seq_num 183) 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:1069: finishing set_chunk to oid 4 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 1421824 length: 1240064 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 96 snap 4 seq_num 96) dirty exists 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:1074: got (22) Invalid argument 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:1066: finishing write tid 1 to smithi10548823-37 2024-04-20T02:51:27.100 INFO:tasks.rados.rados.0.smithi105.stdout:1073: expect (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:51:27.127 INFO:teuthology.orchestra.run.smithi105.stderr:set pool 4 pg_num to 16 2024-04-20T02:51:27.157 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:27.158+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:27.208 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:27.208 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:27.218 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:27.234 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:27.252 INFO:tasks.rados.rados.0.smithi105.stdout:1066: finishing write tid 2 to smithi10548823-37 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1066: finishing write tid 3 to smithi10548823-37 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1061: done (13 left) 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1069: done (12 left) 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1070: done (11 left) 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1073: done (10 left) 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1074: done (9 left) 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1080: done (8 left) 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1081: write oid 5 current snap is 18 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1066: finishing write tid 4 to smithi10548823-37 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1066: oid 37 updating version 0 to 38 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1066: oid 37 version 38 is already newer than 36 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1066: oid 37 version 38 is already newer than 37 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 38 (ObjNum 184 snap 17 seq_num 184) dirty exists 2024-04-20T02:51:27.253 INFO:tasks.rados.rados.0.smithi105.stdout:1066: left oid 37 (ObjNum 184 snap 17 seq_num 184) 2024-04-20T02:51:27.254 INFO:tasks.rados.rados.0.smithi105.stdout:1075: expect (ObjNum 86 snap 2 seq_num 86) 2024-04-20T02:51:27.254 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:27.259 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:27.260+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:27.272 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:27.292 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:27.311 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:27.330 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:27.349 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:27.361 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:27.359+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:27.369 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:27.371 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:27.371 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:27.371 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:27.371 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:27.371 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:27.372 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:27.372 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:27.372 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:27.392 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:27.392 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 67.787901106999996, 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.393 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:51:27.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:27.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:51:27.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:30 4.8 4:1c68e542:::smithi10548823-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 46.027748918999997, 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033039351000000002, 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.400 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 30 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364067+0000", 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364066+0000", 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364312+0000", 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000246708 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364328+0000", 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5376999999999999e-05 2024-04-20T02:51:27.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364339+0000", 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.092e-05 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375692+0000", 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011353103999999999 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375805+0000", 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000113414 2024-04-20T02:51:27.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.375970+0000", 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016508399999999999 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388615+0000", 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012645145 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391156+0000", 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025406410000000002 2024-04-20T02:51:27.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396990+0000", 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0058340390000000001 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397042+0000", 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2454000000000003e-05 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397106+0000", 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3632000000000003e-05 2024-04-20T02:51:27.405 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:31 4.8 4:1c68e542:::smithi10548823-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 46.027462223000001, 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032910542000000001, 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.406 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 31 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364353+0000", 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364357+0000", 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1160000000000001e-06 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364363+0000", 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1680000000000001e-06 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364367+0000", 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8750000000000002e-06 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388814+0000", 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024447437999999998 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.388914+0000", 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9557999999999998e-05 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:27.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389062+0000", 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014799600000000001 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.391160+0000", 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0020979760000000001 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.396038+0000", 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048784689999999999 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397155+0000", 2024-04-20T02:51:27.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001116286 2024-04-20T02:51:27.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397208+0000", 2024-04-20T02:51:27.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3273000000000003e-05 2024-04-20T02:51:27.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397264+0000", 2024-04-20T02:51:27.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5962999999999999e-05 2024-04-20T02:51:27.412 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.412 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.412 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.412 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:32 4.8 4:1c68e542:::smithi10548823-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 46.027443509000001, 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033121840999999999, 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 32 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:27.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364372+0000", 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364375+0000", 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1149999999999998e-06 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364382+0000", 2024-04-20T02:51:27.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4259999999999997e-06 2024-04-20T02:51:27.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.364387+0000", 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6129999999999997e-06 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389813+0000", 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025426621 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:27.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.389905+0000", 2024-04-20T02:51:27.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2011999999999999e-05 2024-04-20T02:51:27.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397299+0000", 2024-04-20T02:51:27.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007393897 2024-04-20T02:51:27.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397395+0000", 2024-04-20T02:51:27.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5293999999999999e-05 2024-04-20T02:51:27.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.397494+0000", 2024-04-20T02:51:27.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9289000000000004e-05 2024-04-20T02:51:27.420 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.420 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.421 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:51:27.422 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:27.422 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 46.024034708000002, 2024-04-20T02:51:27.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:51:27.422 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.422 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.422 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:27.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:51:27.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:51:27.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:27.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:51:27.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:51:27.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:51:27.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:51:27.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:51:27.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:51:27.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:51:27.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:51:27.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:51:27.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:51:27.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:51:27.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:51:27.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:51:27.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:51:27.431 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.431 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.431 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.432 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:27.432 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:27.432 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.910731667, 2024-04-20T02:51:27.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:51:27.432 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.433 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.433 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.433 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.433 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.433 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:51:27.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.433 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:27.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:51:27.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:51:27.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:51:27.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:51:27.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:51:27.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:51:27.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:51:27.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:51:27.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:51:27.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:51:27.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:51:27.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:51:27.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:51:27.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:27.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:51:27.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:51:27.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:27.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:51:27.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:51:27.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:51:27.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:51:27.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:51:27.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:51:27.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:51:27.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:51:27.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:51:27.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:51:27.444 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.444 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.444 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.444 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:66 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:27.444 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:27.444 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.905373779000001, 2024-04-20T02:51:27.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033365526, 2024-04-20T02:51:27.445 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.445 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.445 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.445 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.445 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.445 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 66 2024-04-20T02:51:27.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.445 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:27.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486442+0000", 2024-04-20T02:51:27.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486438+0000", 2024-04-20T02:51:27.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999962 2024-04-20T02:51:27.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486778+0000", 2024-04-20T02:51:27.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000339709 2024-04-20T02:51:27.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486798+0000", 2024-04-20T02:51:27.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0462000000000002e-05 2024-04-20T02:51:27.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486812+0000", 2024-04-20T02:51:27.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3589e-05 2024-04-20T02:51:27.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496889+0000", 2024-04-20T02:51:27.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010077477 2024-04-20T02:51:27.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.496978+0000", 2024-04-20T02:51:27.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8205e-05 2024-04-20T02:51:27.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:27.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.497125+0000", 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000147665 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518444+0000", 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.021318713 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519407+0000", 2024-04-20T02:51:27.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00096312999999999998 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519727+0000", 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031953500000000002 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519764+0000", 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7406000000000001e-05 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519807+0000", 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3402000000000002e-05 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:67 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2268912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.904988594999999, 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033307638000000001, 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.454 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 67 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486827+0000", 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486826+0000", 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486831+0000", 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0189999999999997e-06 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486839+0000", 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4710000000000001e-06 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486844+0000", 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.57e-06 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509240+0000", 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022395241 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509338+0000", 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8034000000000004e-05 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509498+0000", 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015993999999999999 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518471+0000", 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0089732730000000004 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.519859+0000", 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013877130000000001 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520042+0000", 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000182981 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520084+0000", 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2518999999999997e-05 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520135+0000", 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0491000000000002e-05 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.459 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.90496615, 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.460 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:51:27.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.461 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:27.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:27.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:51:27.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:27.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:51:27.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:51:27.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:51:27.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:51:27.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:51:27.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.465 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.550946015000001, 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:51:27.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.540582409999999, 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.470 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:27.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:51:27.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:51:27.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:51:27.475 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.532497931000002, 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:51:27.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:981 4.7 e32/27 4:e9876a63:::smithi10548823-30:head v 32'19, mlcod=30'17)", 2024-04-20T02:51:27.479 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 40.802445456999997, 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033092985999999998, 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589370+0000", 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.589369+0000", 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590134+0000", 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076479 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590148+0000", 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3679999999999999e-05 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590157+0000", 2024-04-20T02:51:27.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:51:27.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590209+0000", 2024-04-20T02:51:27.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2237999999999997e-05 2024-04-20T02:51:27.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.590235+0000", 2024-04-20T02:51:27.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6251e-05 2024-04-20T02:51:27.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622430+0000", 2024-04-20T02:51:27.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032194953999999998 2024-04-20T02:51:27.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:46.622463+0000", 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2780000000000001e-05 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1230 4.5 e38/22 4:a94dc531:::smithi10548823-32:head v 38'11, mlcod=29'8)", 2024-04-20T02:51:27.484 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.979300059000003, 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033312785999999997, 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412516+0000", 2024-04-20T02:51:27.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412515+0000", 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412827+0000", 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031200999999999999 2024-04-20T02:51:27.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412835+0000", 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.61e-06 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412842+0000", 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6990000000000003e-06 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412921+0000", 2024-04-20T02:51:27.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8380000000000005e-05 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.412944+0000", 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3609e-05 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445795+0000", 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032850279000000003 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.445828+0000", 2024-04-20T02:51:27.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3673000000000001e-05 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.250939706, 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.489 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:51:27.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:51:27.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:27.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:51:27.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:51:27.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:51:27.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:51:27.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:27.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:51:27.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:51:27.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:51:27.495 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.245802596000001, 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.496 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:51:27.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:51:27.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.245529584, 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.502 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.503 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:51:27.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.503 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:27.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:51:27.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:51:27.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:51:27.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:51:27.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:51:27.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:51:27.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.245500198999999, 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:27.509 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:51:27.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:27.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:51:27.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:51:27.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:51:27.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:51:27.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:51:27.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:51:27.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:51:27.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:51:27.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:27.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:51:27.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:51:27.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:51:27.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:51:27.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:51:27.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.936606233999999, 2024-04-20T02:51:27.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035060916999999997, 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456603+0000", 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001394295 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456616+0000", 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2605999999999999e-05 2024-04-20T02:51:27.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456631+0000", 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4735e-05 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456682+0000", 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1048000000000003e-05 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456705+0000", 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2881e-05 2024-04-20T02:51:27.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490235+0000", 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033530250999999997 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490270+0000", 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5379000000000001e-05 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.5581728999999997, 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:27.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:27.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.524 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:27.525 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:27.525 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:27.526 INFO:tasks.rados.rados.0.smithi105.stdout:1071: expect (ObjNum 181 snap 15 seq_num 181) 2024-04-20T02:51:27.527 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:27.527 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:27.526+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:27.590 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:51:27.599 INFO:tasks.rados.rados.0.smithi105.stdout:1072: got (0) Success 2024-04-20T02:51:27.599 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 147 (ObjNum 188079136 snap 32641 seq_num 304) clean dne 2024-04-20T02:51:27.599 INFO:tasks.rados.rados.0.smithi105.stdout:1078: got (22) Invalid argument 2024-04-20T02:51:27.599 INFO:tasks.rados.rados.0.smithi105.stdout:1077: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:27.627 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:27.627+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:27.657 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:27.657 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:27.657 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:27.657 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:27.667 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:27.716 INFO:tasks.rados.rados.0.smithi105.stdout:1081: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-04-20T02:51:27.728 INFO:tasks.rados.rados.0.smithi105.stdout:1081: writing smithi10548823-5 from 726102 to 1514888 tid 1 2024-04-20T02:51:27.731 INFO:tasks.rados.rados.0.smithi105.stdout:1081: writing smithi10548823-5 from 1917886 to 2528279 tid 2 2024-04-20T02:51:27.736 INFO:tasks.rados.rados.0.smithi105.stdout:1081: writing smithi10548823-5 from 3108206 to 3126102 tid 3 2024-04-20T02:51:27.737 INFO:tasks.rados.rados.0.smithi105.stdout:1066: done (8 left) 2024-04-20T02:51:27.737 INFO:tasks.rados.rados.0.smithi105.stdout:1071: done (7 left) 2024-04-20T02:51:27.737 INFO:tasks.rados.rados.0.smithi105.stdout:1072: done (6 left) 2024-04-20T02:51:27.737 INFO:tasks.rados.rados.0.smithi105.stdout:1075: done (5 left) 2024-04-20T02:51:27.737 INFO:tasks.rados.rados.0.smithi105.stdout:1077: done (4 left) 2024-04-20T02:51:27.737 INFO:tasks.rados.rados.0.smithi105.stdout:1078: done (3 left) 2024-04-20T02:51:27.738 INFO:tasks.rados.rados.0.smithi105.stdout:1082: read oid 12 snap -1 2024-04-20T02:51:27.738 INFO:tasks.rados.rados.0.smithi105.stdout:1082: expect (ObjNum 167 snap 14 seq_num 167) 2024-04-20T02:51:27.738 INFO:tasks.rados.rados.0.smithi105.stdout:1083: tier_evict oid 45 2024-04-20T02:51:27.738 INFO:tasks.rados.rados.0.smithi105.stdout:1083: tier_evict oid 45 snap -1 2024-04-20T02:51:27.738 INFO:tasks.rados.rados.0.smithi105.stdout:1079: finishing set_chunk to oid 35 2024-04-20T02:51:27.738 INFO:tasks.rados.rados.0.smithi105.stdout:1076: finishing write tid 1 to smithi10548823-30 2024-04-20T02:51:27.739 INFO:tasks.rados.rados.0.smithi105.stdout:1076: finishing write tid 2 to smithi10548823-30 2024-04-20T02:51:27.739 INFO:tasks.rados.rados.0.smithi105.stdout:1076: finishing write tid 3 to smithi10548823-30 2024-04-20T02:51:27.739 INFO:tasks.rados.rados.0.smithi105.stdout:1076: finishing write tid 4 to smithi10548823-30 2024-04-20T02:51:27.739 INFO:tasks.rados.rados.0.smithi105.stdout:1076: oid 30 updating version 0 to 50 2024-04-20T02:51:27.739 INFO:tasks.rados.rados.0.smithi105.stdout:1076: oid 30 version 50 is already newer than 49 2024-04-20T02:51:27.739 INFO:tasks.rados.rados.0.smithi105.stdout:1076: oid 30 updating version 50 to 51 2024-04-20T02:51:27.740 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 51 (ObjNum 185 snap 17 seq_num 185) dirty exists 2024-04-20T02:51:27.740 INFO:tasks.rados.rados.0.smithi105.stdout:1076: left oid 30 (ObjNum 185 snap 17 seq_num 185) 2024-04-20T02:51:27.740 INFO:tasks.rados.rados.0.smithi105.stdout:1076: done (4 left) 2024-04-20T02:51:27.740 INFO:tasks.rados.rados.0.smithi105.stdout:1079: done (3 left) 2024-04-20T02:51:27.740 INFO:tasks.rados.rados.0.smithi105.stdout:1084: read oid 24 snap -1 2024-04-20T02:51:27.740 INFO:tasks.rados.rados.0.smithi105.stdout:1084: expect (ObjNum 180 snap 15 seq_num 180) 2024-04-20T02:51:27.740 INFO:tasks.rados.rados.0.smithi105.stdout:1085: write oid 43 current snap is 18 2024-04-20T02:51:27.741 INFO:tasks.rados.rados.0.smithi105.stdout:1085: seq_num 187 ranges {742909=464035,1976368=754051} 2024-04-20T02:51:27.742 INFO:tasks.rados.rados.0.smithi105.stdout:1085: writing smithi10548823-43 from 742909 to 1206944 tid 1 2024-04-20T02:51:27.750 INFO:tasks.rados.rados.0.smithi105.stdout:1085: writing smithi10548823-43 from 1976368 to 2730419 tid 2 2024-04-20T02:51:27.751 INFO:tasks.rados.rados.0.smithi105.stdout:1086: set_chunk oid 10 target oid 2024-04-20T02:51:27.751 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:27.752+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:27.807 INFO:tasks.rados.rados.0.smithi105.stdout:1086: set_chunk oid 10 offset: 148992 length: 2165760 target oid 7146376437549e0b99264d159bf1e9c8bc43ac1c7eb519f65899cfd592a43a4a offset: 0 snap -1 2024-04-20T02:51:27.808 INFO:tasks.rados.rados.0.smithi105.stdout:1087: read oid 22 snap -1 2024-04-20T02:51:27.808 INFO:tasks.rados.rados.0.smithi105.stdout:1087: expect (ObjNum 163 snap 14 seq_num 163) 2024-04-20T02:51:27.808 INFO:tasks.rados.rados.0.smithi105.stdout:1088: write (excl) oid 23 current snap is 18 2024-04-20T02:51:27.808 INFO:tasks.rados.rados.0.smithi105.stdout:1088: seq_num 188 ranges {759716=539284,2034850=614062} 2024-04-20T02:51:27.814 INFO:tasks.rados.rados.0.smithi105.stdout:1088: writing smithi10548823-23 from 759716 to 1299000 tid 1 2024-04-20T02:51:27.822 INFO:tasks.rados.rados.0.smithi105.stdout:1088: writing smithi10548823-23 from 2034850 to 2648912 tid 2 2024-04-20T02:51:27.823 INFO:tasks.rados.rados.0.smithi105.stdout:1089: tier_flush oid 2 2024-04-20T02:51:27.823 INFO:tasks.rados.rados.0.smithi105.stdout:1089: tier_flush oid 2 snap -1 2024-04-20T02:51:27.823 INFO:tasks.rados.rados.0.smithi105.stdout:1083: got (22) Invalid argument 2024-04-20T02:51:27.823 INFO:tasks.rados.rados.0.smithi105.stdout:1083: done (8 left) 2024-04-20T02:51:27.823 INFO:tasks.rados.rados.0.smithi105.stdout:1090: read oid 32 snap -1 2024-04-20T02:51:27.823 INFO:tasks.rados.rados.0.smithi105.stdout:1090: expect (ObjNum 183 snap 17 seq_num 183) 2024-04-20T02:51:27.823 INFO:tasks.rados.rados.0.smithi105.stdout:1091: write (excl) oid 33 current snap is 18 2024-04-20T02:51:27.823 INFO:tasks.rados.rados.0.smithi105.stdout:1082: expect (ObjNum 167 snap 14 seq_num 167) 2024-04-20T02:51:27.829 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:27.830+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:27.847 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:27.847 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:27.847 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:27.847 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:27.847 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:27.858 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:27.930 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:27.931+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:27.977 INFO:tasks.rados.rados.0.smithi105.stdout:1081: finishing write tid 1 to smithi10548823-5 2024-04-20T02:51:27.977 INFO:tasks.rados.rados.0.smithi105.stdout:1084: expect (ObjNum 180 snap 15 seq_num 180) 2024-04-20T02:51:28.029 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:28.029 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:28.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:28.029 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:28.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.029 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:152 3.2 e48/47 3:54f0c00e:::2528189ec1c891f179c539fa23af2c87b9e8d2c19f70e70fac8934d3cc51ccee:head v 48'96, mlcod=0'0)", 2024-04-20T02:51:28.030 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:28.030 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.5954202, 2024-04-20T02:51:28.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020196797999999998, 2024-04-20T02:51:28.030 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.030 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.030 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:28.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:28.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433091+0000", 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000185589 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433102+0000", 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433108+0000", 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5660000000000003e-06 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439137+0000", 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060284329999999997 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439166+0000", 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9278e-05 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453075+0000", 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013909163 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453103+0000", 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7725999999999998e-05 2024-04-20T02:51:28.034 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:154 3.2 e48/47 3:7d72d9ce:::7b4ada8765f224bb4a516820a83f51a402b4ee671c5f45bb3fbe5e6656cf1f14:head v 48'97, mlcod=42'91)", 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.591514568000001, 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016320239, 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436811+0000", 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437600+0000", 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078939699999999995 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437613+0000", 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2588e-05 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437624+0000", 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0542e-05 2024-04-20T02:51:28.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442541+0000", 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049178809999999998 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442565+0000", 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3558e-05 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453110+0000", 2024-04-20T02:51:28.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010545172 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453132+0000", 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1929e-05 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:155 3.2 e48/47 3:715fab8c:::b945a2927747be608263bdbe1fe814f665c73382cb3375555da5aa95a92567ff:head v 48'99, mlcod=42'91)", 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.588727512999998, 2024-04-20T02:51:28.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017321785999999999, 2024-04-20T02:51:28.040 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.040 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.040 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:28.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:28.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439598+0000", 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439954+0000", 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00035586299999999999 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439970+0000", 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5889e-05 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439981+0000", 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0754000000000001e-05 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443643+0000", 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036621259999999999 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443671+0000", 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8289e-05 2024-04-20T02:51:28.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456885+0000", 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013213964999999999 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456921+0000", 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.044 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1635 3.2 e51/49 3:67667c16:::7aea211498f946c134ed12b1f5632511a2f083a0eb2114d1051f38dc7326b113:head v 51'100, mlcod=0'0)", 2024-04-20T02:51:28.045 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:28.045 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.180836566, 2024-04-20T02:51:28.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019392829, 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:28.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847489+0000", 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850819+0000", 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033292069999999998 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850831+0000", 2024-04-20T02:51:28.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2320000000000001e-05 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850838+0000", 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2980000000000003e-06 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850874+0000", 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5800999999999998e-05 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850890+0000", 2024-04-20T02:51:28.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5536000000000002e-05 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866849+0000", 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015959346999999999 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866883+0000", 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3776999999999999e-05 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.579688327, 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1665 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'78, mlcod=50'73)", 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.505088474000001, 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025918594, 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.054 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523236+0000", 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:28.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523658+0000", 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000422082 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523676+0000", 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7716000000000001e-05 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523686+0000", 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9790000000000005e-06 2024-04-20T02:51:28.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523734+0000", 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7579e-05 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523759+0000", 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5593e-05 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549122+0000", 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025363087999999999 2024-04-20T02:51:28.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549157+0000", 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4242999999999997e-05 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.573180095000001, 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034212370999999998, 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.058 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:28.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456260+0000", 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011140480000000001 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456273+0000", 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456280+0000", 2024-04-20T02:51:28.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1130000000000003e-06 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456331+0000", 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1446999999999997e-05 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456357+0000", 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6001999999999999e-05 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489309+0000", 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032951972000000003 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489359+0000", 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9552000000000003e-05 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1775 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'41, mlcod=53'38)", 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.274798605000001, 2024-04-20T02:51:28.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038672274999999999, 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754868+0000", 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013404619999999999 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754881+0000", 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.323e-05 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754892+0000", 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1154e-05 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754967+0000", 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4326999999999995e-05 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.755006+0000", 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9338000000000002e-05 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792154+0000", 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037147924999999998 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792200+0000", 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6044999999999997e-05 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.068 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1776 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'42, mlcod=53'38)", 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.254818427, 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019186657999999999, 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:28.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773507+0000", 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774271+0000", 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076432799999999999 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774284+0000", 2024-04-20T02:51:28.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2483000000000001e-05 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774294+0000", 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6569999999999992e-06 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774338+0000", 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4576000000000001e-05 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774358+0000", 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9449999999999998e-05 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792664+0000", 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018306413000000001 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792695+0000", 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0685000000000002e-05 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1822 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'94, mlcod=55'92)", 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.924776472, 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031823907999999998, 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103786+0000", 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000236658 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103797+0000", 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0951000000000001e-05 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103804+0000", 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0960000000000003e-06 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103860+0000", 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5781e-05 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103887+0000", 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6999999999999999e-05 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135333+0000", 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031446004 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135374+0000", 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0778999999999997e-05 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.076 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1823 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'95, mlcod=55'92)", 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.910289466, 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017980995999999999, 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118036+0000", 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118363+0000", 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00032734600000000001 2024-04-20T02:51:28.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118381+0000", 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7816e-05 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118391+0000", 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0155e-05 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119591+0000", 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011995560000000001 2024-04-20T02:51:28.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119618+0000", 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7475e-05 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135988+0000", 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016370096000000001 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.136018+0000", 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9745999999999999e-05 2024-04-20T02:51:28.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1836 4.1 e58/46 4:8b927dd6:::smithi10548823-28:head v 58'99, mlcod=56'96)", 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.542760464000001, 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030374581000000001, 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:28.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485930+0000", 2024-04-20T02:51:28.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00036455800000000002 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485941+0000", 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1017e-05 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485950+0000", 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.85e-06 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486000+0000", 2024-04-20T02:51:28.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9589999999999998e-05 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486025+0000", 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.529e-05 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515909+0000", 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029884131000000001 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515941+0000", 2024-04-20T02:51:28.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1569000000000002e-05 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5295055600000005, 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.085 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:28.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:51:28.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:51:28.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5294401109999995, 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:51:28.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:51:28.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:28.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:51:28.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:51:28.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.094 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5294146130000001, 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:28.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:51:28.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:51:28.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.099 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1891 4.17 e61/60 4:e9876a63:::smithi10548823-30:head v 61'45, mlcod=0'0)", 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.514868817, 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018777245000000001, 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:28.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513457+0000", 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514596+0000", 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011393289999999999 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514612+0000", 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6311e-05 2024-04-20T02:51:28.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514623+0000", 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056999999999999e-05 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514670+0000", 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6557999999999997e-05 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514698+0000", 2024-04-20T02:51:28.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8121000000000001e-05 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.532204+0000", 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017505969999999999 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.532235+0000", 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0731000000000001e-05 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.103 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1892 4.17 e61/60 4:e9876a63:::smithi10548823-30:head v 61'46, mlcod=0'0)", 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.4915298329999995, 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020303921999999999, 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536796+0000", 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537578+0000", 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078217499999999997 2024-04-20T02:51:28.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537592+0000", 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3721e-05 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537602+0000", 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0380000000000001e-05 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537638+0000", 2024-04-20T02:51:28.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5559000000000003e-05 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537657+0000", 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9242e-05 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.557064+0000", 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019406970999999999 2024-04-20T02:51:28.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.557101+0000", 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6572000000000002e-05 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2004 4.13 e65/63 4:cfd1e4d4:::smithi10548823-9:head v 65'159, mlcod=64'156)", 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1801895650000001, 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036995865000000003, 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.108 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849186+0000", 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010492889999999999 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849197+0000", 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1365999999999999e-05 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849204+0000", 2024-04-20T02:51:28.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1069999999999999e-06 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849250+0000", 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5213999999999998e-05 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849279+0000", 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9149000000000002e-05 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885093+0000", 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035813921999999998 2024-04-20T02:51:28.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885133+0000", 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0077999999999998e-05 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2005 4.13 e65/63 4:cfd1e4d4:::smithi10548823-9:head v 65'160, mlcod=64'156)", 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.1603285979999995, 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.01736478, 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.112 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867997+0000", 2024-04-20T02:51:28.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868547+0000", 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000550287 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868563+0000", 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5676e-05 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868572+0000", 2024-04-20T02:51:28.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0440000000000001e-06 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.874844+0000", 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0062724460000000001 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.874865+0000", 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0712000000000001e-05 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885334+0000", 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010468691 2024-04-20T02:51:28.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885363+0000", 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8923000000000001e-05 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2029 3.2 e66/49 3:70b121b2:::b13b8a8d7724a2b6951a53e14e5e568e1aa45f9c18b0373ab4a92ed71f345848:head v 66'106, mlcod=53'104)", 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:24.119464+0000", 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 3.9088628810000001, 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023218222, 2024-04-20T02:51:28.116 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.119464+0000", 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.119464+0000", 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:28.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.119462+0000", 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122123+0000", 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002661629 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122140+0000", 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6290000000000002e-05 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122147+0000", 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.695e-06 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122201+0000", 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.393e-05 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122228+0000", 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6367999999999999e-05 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:28.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.142651+0000", 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020423884 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.142682+0000", 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0301999999999999e-05 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:28.120 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1081: finishing write tid 2 to smithi10548823-5 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1081: finishing write tid 3 to smithi10548823-5 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1081: finishing write tid 4 to smithi10548823-5 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1081: finishing write tid 5 to smithi10548823-5 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1081: oid 5 updating version 0 to 116 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1081: oid 5 version 116 is already newer than 114 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1081: oid 5 version 116 is already newer than 115 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1081: oid 5 version 116 is already newer than 113 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 116 (ObjNum 186 snap 18 seq_num 186) dirty exists 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1081: left oid 5 (ObjNum 186 snap 18 seq_num 186) 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1085: finishing write tid 1 to smithi10548823-43 2024-04-20T02:51:28.122 INFO:tasks.rados.rados.0.smithi105.stdout:1085: finishing write tid 2 to smithi10548823-43 2024-04-20T02:51:28.123 INFO:tasks.rados.rados.0.smithi105.stdout:1085: finishing write tid 3 to smithi10548823-43 2024-04-20T02:51:28.123 INFO:tasks.rados.rados.0.smithi105.stdout:1085: finishing write tid 4 to smithi10548823-43 2024-04-20T02:51:28.123 INFO:tasks.rados.rados.0.smithi105.stdout:1085: oid 43 updating version 0 to 76 2024-04-20T02:51:28.123 INFO:tasks.rados.rados.0.smithi105.stdout:1085: oid 43 updating version 76 to 78 2024-04-20T02:51:28.123 INFO:tasks.rados.rados.0.smithi105.stdout:1085: oid 43 version 78 is already newer than 77 2024-04-20T02:51:28.123 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 78 (ObjNum 187 snap 18 seq_num 187) dirty exists 2024-04-20T02:51:28.123 INFO:tasks.rados.rados.0.smithi105.stdout:1085: left oid 43 (ObjNum 187 snap 18 seq_num 187) 2024-04-20T02:51:28.123 INFO:tasks.rados.rados.0.smithi105.stdout:1091: seq_num 189 ranges {776523=614533,1976979=757720} 2024-04-20T02:51:28.123 INFO:tasks.rados.rados.0.smithi105.stdout:1091: writing smithi10548823-33 from 776523 to 1391056 tid 1 2024-04-20T02:51:28.123 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:28.122+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:28.124 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:28.131 INFO:tasks.rados.rados.0.smithi105.stdout:1091: writing smithi10548823-33 from 1976979 to 2734699 tid 2 2024-04-20T02:51:28.131 INFO:tasks.rados.rados.0.smithi105.stdout:1081: done (9 left) 2024-04-20T02:51:28.131 INFO:tasks.rados.rados.0.smithi105.stdout:1082: done (8 left) 2024-04-20T02:51:28.132 INFO:tasks.rados.rados.0.smithi105.stdout:1084: done (7 left) 2024-04-20T02:51:28.132 INFO:tasks.rados.rados.0.smithi105.stdout:1085: done (6 left) 2024-04-20T02:51:28.132 INFO:tasks.rados.rados.0.smithi105.stdout:1092: tier_promote oid 36 2024-04-20T02:51:28.132 INFO:tasks.rados.rados.0.smithi105.stdout:1087: expect (ObjNum 163 snap 14 seq_num 163) 2024-04-20T02:51:28.138 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:51:28.223 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:28.223+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:28.266 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:28.266 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:28.266 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:28.266 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:28.267 INFO:tasks.rados.rados.0.smithi105.stdout:1086: finishing set_chunk to oid 10 2024-04-20T02:51:28.267 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 96 (ObjNum 156 snap 10 seq_num 156) dirty exists 2024-04-20T02:51:28.267 INFO:tasks.rados.rados.0.smithi105.stdout:1089: got (16) Device or resource busy 2024-04-20T02:51:28.267 INFO:tasks.rados.rados.0.smithi105.stdout:check_oldest_snap_flushed oid 2 v 88 (ObjNum 155 snap 10 seq_num 155) unflushed 11 2024-04-20T02:51:28.267 INFO:tasks.rados.rados.0.smithi105.stdout:1090: expect (ObjNum 183 snap 17 seq_num 183) 2024-04-20T02:51:28.277 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:28.324 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:28.324+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:28.405 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:28.405 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:28.415 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:28.457 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:28.457+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:28.481 INFO:tasks.rados.rados.0.smithi105.stdout:1088: finishing write tid 1 to smithi10548823-23 2024-04-20T02:51:28.481 INFO:tasks.rados.rados.0.smithi105.stdout:1088: finishing write tid 2 to smithi10548823-23 2024-04-20T02:51:28.481 INFO:tasks.rados.rados.0.smithi105.stdout:1086: done (6 left) 2024-04-20T02:51:28.481 INFO:tasks.rados.rados.0.smithi105.stdout:1087: done (5 left) 2024-04-20T02:51:28.481 INFO:tasks.rados.rados.0.smithi105.stdout:1089: done (4 left) 2024-04-20T02:51:28.481 INFO:tasks.rados.rados.0.smithi105.stdout:1090: done (3 left) 2024-04-20T02:51:28.481 INFO:tasks.rados.rados.0.smithi105.stdout:1093: rollback oid 49 current snap is 18 2024-04-20T02:51:28.481 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 49 to 16 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1088: finishing write tid 3 to smithi10548823-23 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1088: finishing write tid 4 to smithi10548823-23 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1088: oid 23 updating version 0 to 67 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1088: oid 23 version 67 is already newer than 65 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1088: oid 23 version 67 is already newer than 66 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 67 (ObjNum 188 snap 18 seq_num 188) dirty exists 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1088: left oid 23 (ObjNum 188 snap 18 seq_num 188) 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1091: finishing write tid 1 to smithi10548823-33 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1091: finishing write tid 2 to smithi10548823-33 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1091: finishing write tid 3 to smithi10548823-33 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1091: finishing write tid 4 to smithi10548823-33 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1091: oid 33 updating version 0 to 43 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1091: oid 33 updating version 43 to 44 2024-04-20T02:51:28.482 INFO:tasks.rados.rados.0.smithi105.stdout:1091: oid 33 updating version 44 to 45 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 45 (ObjNum 189 snap 18 seq_num 189) dirty exists 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:1091: left oid 33 (ObjNum 189 snap 18 seq_num 189) 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:1092: got (0) Success 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 162 (ObjNum 178 snap 15 seq_num 178) dirty exists 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:1088: done (3 left) 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:1091: done (2 left) 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:1092: done (1 left) 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:1094: tier_promote oid 2 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:1095: rollback oid 40 current snap is 18 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 40 to 12 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:1096: read oid 13 snap -1 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:1096: expect (ObjNum 151 snap 10 seq_num 151) 2024-04-20T02:51:28.483 INFO:tasks.rados.rados.0.smithi105.stdout:1097: set_chunk oid 1 target oid 2024-04-20T02:51:28.526 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:28.526+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:927 4.3 4:cf11ccc0:::smithi10548823-4:head [write 3121996~104915] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.210367067, 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058986447999999997, 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.589 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 927 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377548+0000", 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377607+0000", 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9413999999999997e-05 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377616+0000", 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4880000000000007e-06 2024-04-20T02:51:28.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377621+0000", 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.682e-06 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407778+0000", 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030156110999999999 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407866+0000", 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8645000000000003e-05 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407985+0000", 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118874 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411741+0000", 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037563060000000001 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411854+0000", 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112898 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436438+0000", 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024583885 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436483+0000", 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5334999999999999e-05 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436535+0000", 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1780000000000002e-05 2024-04-20T02:51:28.594 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:928 4.3 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (55) in=7b,truncate 3226911] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.210288460999998, 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059064512999999999, 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 928 2024-04-20T02:51:28.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377630+0000", 2024-04-20T02:51:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7259999999999998e-06 2024-04-20T02:51:28.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377633+0000", 2024-04-20T02:51:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1489999999999998e-06 2024-04-20T02:51:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377636+0000", 2024-04-20T02:51:28.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7760000000000002e-06 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410754+0000", 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033117944000000003 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410850+0000", 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6522999999999997e-05 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411008+0000", 2024-04-20T02:51:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158157 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436151+0000", 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025142813 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436160+0000", 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1460000000000002e-06 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436588+0000", 2024-04-20T02:51:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00042776499999999998 2024-04-20T02:51:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436631+0000", 2024-04-20T02:51:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3084000000000003e-05 2024-04-20T02:51:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436692+0000", 2024-04-20T02:51:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0829000000000003e-05 2024-04-20T02:51:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:929 4.3 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.210277320000003, 2024-04-20T02:51:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059214058, 2024-04-20T02:51:28.603 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 929 2024-04-20T02:51:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377638+0000", 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377640+0000", 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5430000000000001e-06 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377643+0000", 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0900000000000001e-06 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377647+0000", 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2230000000000001e-06 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411271+0000", 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033623635999999998 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411329+0000", 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8043e-05 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436728+0000", 2024-04-20T02:51:28.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025399300999999999 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436793+0000", 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5041000000000003e-05 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436853+0000", 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9446e-05 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:936 4.3 4:ca036f62:::smithi10548823-34:head [write 443718~615603] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.193241987, 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060880228000000002, 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 936 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394672+0000", 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394825+0000", 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015343299999999999 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394844+0000", 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8414999999999999e-05 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394859+0000", 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4738e-05 2024-04-20T02:51:28.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411879+0000", 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017020469999999999 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411954+0000", 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5239000000000002e-05 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412104+0000", 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014944399999999999 2024-04-20T02:51:28.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436402+0000", 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024297828 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436409+0000", 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1779999999999997e-06 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455390+0000", 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018981221 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455508+0000", 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117556 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455554+0000", 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6570000000000003e-05 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:937 4.3 4:ca036f62:::smithi10548823-34:head [write 1564138~470341] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:28.614 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.187903657, 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060119655000000001, 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 937 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400010+0000", 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400222+0000", 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000211208 2024-04-20T02:51:28.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400238+0000", 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419e-05 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400250+0000", 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2091e-05 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425851+0000", 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025600795999999999 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425965+0000", 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011411799999999999 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426134+0000", 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168517 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436937+0000", 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01080328 2024-04-20T02:51:28.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455351+0000", 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018414607 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460001+0000", 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046491809999999996 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460086+0000", 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5066000000000002e-05 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460132+0000", 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6174999999999998e-05 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:938 4.3 4:ca036f62:::smithi10548823-34:head [write 2633066~610652] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.181138201000003, 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054360654000000001, 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 938 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406775+0000", 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406947+0000", 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171619 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406962+0000", 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5248999999999999e-05 2024-04-20T02:51:28.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406979+0000", 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7334999999999999e-05 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436954+0000", 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029974218 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437007+0000", 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3553e-05 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437110+0000", 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455605+0000", 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018494295000000001 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455829+0000", 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022443100000000001 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461044+0000", 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052150499999999997 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461090+0000", 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6100999999999998e-05 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461138+0000", 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8003000000000001e-05 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.625 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:939 4.3 4:ca036f62:::smithi10548823-34:head [setxattr _header (56) in=7b,truncate 3243718] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.180925889999997, 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054380381999999998, 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 939 2024-04-20T02:51:28.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406989+0000", 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406993+0000", 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9400000000000004e-06 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406999+0000", 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4389999999999999e-06 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407008+0000", 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3190000000000001e-06 2024-04-20T02:51:28.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.454982+0000", 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047973729999999999 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455051+0000", 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9197999999999997e-05 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455157+0000", 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106302 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456120+0000", 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096236599999999998 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456272+0000", 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000152573 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461274+0000", 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005002055 2024-04-20T02:51:28.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461318+0000", 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3282999999999999e-05 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461370+0000", 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2704999999999998e-05 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.631 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:940 4.3 4:ca036f62:::smithi10548823-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.180901341000002, 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054543401999999998, 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 940 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407014+0000", 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407017+0000", 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9790000000000002e-06 2024-04-20T02:51:28.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407022+0000", 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5539999999999997e-06 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407027+0000", 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6099999999999999e-06 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455643+0000", 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.048616331999999998 2024-04-20T02:51:28.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455714+0000", 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0610000000000003e-05 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461403+0000", 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056888219999999996 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461480+0000", 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7342000000000006e-05 2024-04-20T02:51:28.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461558+0000", 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7780000000000004e-05 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.043255805999998, 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.636 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:28.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:51:28.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.655333345000003, 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.640 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.645 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.655073139000002, 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:28.651 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.655051671000003, 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:28.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.461379739000002, 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:51:28.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:133 3.1 3:8efdf53d:::0188443c5b8894a8247535d52a547bece3a4c5f490fb220852b80e77d71ddcda:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.460655216999999, 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060974049000000002, 2024-04-20T02:51:28.661 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127260+0000", 2024-04-20T02:51:28.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127435+0000", 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017459800000000001 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127443+0000", 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3790000000000006e-06 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127450+0000", 2024-04-20T02:51:28.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5089999999999999e-06 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.129973+0000", 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025229639999999999 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130090+0000", 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117521 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130323+0000", 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023271199999999999 2024-04-20T02:51:28.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.147581+0000", 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017258440999999999 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155339+0000", 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077577189999999997 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188139+0000", 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032800425000000001 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188195+0000", 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5773999999999998e-05 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188235+0000", 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9427e-05 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:136 3.1 3:a965252c:::a490375a0954219328244dfa32cf9967dfac8ed04667cb9b5ce15f4910db2ee6:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:28.666 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.460460000000001, 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060878320999999999, 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127455+0000", 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128197+0000", 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074176999999999999 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128208+0000", 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0498e-05 2024-04-20T02:51:28.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128216+0000", 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2630000000000003e-06 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135797+0000", 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0075808669999999998 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135930+0000", 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013299000000000001 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.136192+0000", 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026213899999999999 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148053+0000", 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011860723 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155346+0000", 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0072930039999999996 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188270+0000", 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032924299999999997 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188298+0000", 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8041000000000001e-05 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188334+0000", 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6247e-05 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.671 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:139 3.1 3:90772484:::f9b8a83c02fab1c87f6aef9574ab68b1d6a75b657e100bac3693f85cac340cb2:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.453802074999999, 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054361675999999998, 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:28.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134113+0000", 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134645+0000", 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053281300000000004 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134662+0000", 2024-04-20T02:51:28.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6503000000000001e-05 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134672+0000", 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8090000000000004e-06 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148073+0000", 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013401091 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148177+0000", 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104344 2024-04-20T02:51:28.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148417+0000", 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023968899999999999 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155385+0000", 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0069679260000000002 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155725+0000", 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034046800000000002 2024-04-20T02:51:28.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188383+0000", 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032658154000000002 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188433+0000", 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0059999999999998e-05 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188475+0000", 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2049999999999999e-05 2024-04-20T02:51:28.677 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:140 3.1 3:b80bf598:::82930be3d6fd0d013eae1f4ab54a437986489fd06a4f4934c2432c14687252ab:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.453236820000001, 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053882434, 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134678+0000", 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134756+0000", 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7527999999999994e-05 2024-04-20T02:51:28.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134763+0000", 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.092e-06 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0290000000000002e-06 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152310+0000", 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017540581999999999 2024-04-20T02:51:28.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152431+0000", 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012166 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152621+0000", 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000189944 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155992+0000", 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033703470000000001 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160642+0000", 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046505239999999996 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188508+0000", 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027866134000000001 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188534+0000", 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5945999999999999e-05 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188561+0000", 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7300999999999998e-05 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1799 4.6 4:697da67d:::smithi10548823-48:head [write 588278~715406,stat] snapc 13=[13,d] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.514311664999999, 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053216977999999998, 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.683 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1799 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073603+0000", 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073742+0000", 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000138727 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073754+0000", 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1970000000000001e-05 2024-04-20T02:51:28.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073761+0000", 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.883e-06 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073806+0000", 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5840999999999997e-05 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073872+0000", 2024-04-20T02:51:28.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5520999999999998e-05 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074005+0000", 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013345499999999999 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124313+0000", 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050307628 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124361+0000", 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7608999999999998e-05 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126705+0000", 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0023443790000000002 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126770+0000", 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4737e-05 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126821+0000", 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1375000000000002e-05 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.688 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1801 4.6 4:697da67d:::smithi10548823-48:head [setxattr _header (57) in=7b,truncate 2588278] snapc 13=[13,d] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.508136712000001, 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053362723000000001, 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1801 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079782+0000", 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7840000000000001e-06 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079790+0000", 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3270000000000003e-06 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079799+0000", 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5040000000000008e-06 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123583+0000", 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043783913000000001 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123714+0000", 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013130699999999999 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123889+0000", 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017436200000000001 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132488+0000", 2024-04-20T02:51:28.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085989889999999996 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132702+0000", 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021448600000000001 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133026+0000", 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032330500000000002 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133079+0000", 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3127000000000001e-05 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133142+0000", 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3150000000000004e-05 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1802 4.6 4:697da67d:::smithi10548823-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.508103027000001, 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053694407999999999, 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1802 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079812+0000", 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079815+0000", 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1329999999999999e-06 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079823+0000", 2024-04-20T02:51:28.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.401e-06 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079828+0000", 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4149999999999998e-06 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124387+0000", 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044558788000000002 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124468+0000", 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1447999999999997e-05 2024-04-20T02:51:28.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133186+0000", 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0087175610000000004 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133344+0000", 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158393 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133507+0000", 2024-04-20T02:51:28.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000162829 2024-04-20T02:51:28.699 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.699 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.699 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.699 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:28.699 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:28.700 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:28.701 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:28.701+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:28.801 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:28.801+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:28.869 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:28.870 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:28.870 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:28.870 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:28.882 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:28.926 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:28.926+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:29.003 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:29.003+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:29.009 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:29.010 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:29.010 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:29.010 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:29.010 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:29.020 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:29.105 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:29.106+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:29.158 INFO:tasks.rados.rados.0.smithi105.stdout:1097: set_chunk oid 1 offset: 1285632 length: 868864 target oid f6babd2157cea34b0aa9091a40f11fb5865e3fd976572135c04636cf1ecfb424 offset: 0 snap -1 2024-04-20T02:51:29.158 INFO:tasks.rados.rados.0.smithi105.stdout:1098: read oid 34 snap -1 2024-04-20T02:51:29.158 INFO:tasks.rados.rados.0.smithi105.stdout:1098: expect (ObjNum 170 snap 14 seq_num 170) 2024-04-20T02:51:29.159 INFO:tasks.rados.rados.0.smithi105.stdout:1099: read oid 20 snap -1 2024-04-20T02:51:29.159 INFO:tasks.rados.rados.0.smithi105.stdout:1099: expect (ObjNum 165 snap 14 seq_num 165) 2024-04-20T02:51:29.159 INFO:tasks.rados.rados.0.smithi105.stdout:1100: read oid 3 snap -1 2024-04-20T02:51:29.159 INFO:tasks.rados.rados.0.smithi105.stdout:1100: expect deleted 2024-04-20T02:51:29.159 INFO:tasks.rados.rados.0.smithi105.stdout:1101: tier_promote oid 27 2024-04-20T02:51:29.159 INFO:tasks.rados.rados.0.smithi105.stdout:1096: expect (ObjNum 151 snap 10 seq_num 151) 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.620029590000001, 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.206 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:51:29.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:51:29.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:51:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.604011311999997, 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:29.212 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:51:29.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:51:29.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.603858082999999, 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:29.218 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:51:29.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:51:29.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:51:29.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:51:29.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.603748074000002, 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:51:29.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:51:29.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:51:29.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:51:29.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:51:29.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.603717697999997, 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:29.230 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:51:29.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:51:29.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:51:29.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.599300278999998, 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:29.235 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:51:29.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:51:29.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:51:29.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.590048527999997, 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:29.241 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:29.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.587526384999997, 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.587441517999999, 2024-04-20T02:51:29.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:51:29.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:51:29.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:51:29.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:51:29.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:51:29.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:51:29.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.587423907999998, 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:29.260 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:29.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.265 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:991 4.b e32/22 4:d4308fef:::smithi10548823-6:head v 32'29, mlcod=30'27)", 2024-04-20T02:51:29.265 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:29.265 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.480519653999998, 2024-04-20T02:51:29.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046400134000000003, 2024-04-20T02:51:29.265 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.265 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.265 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724649+0000", 2024-04-20T02:51:29.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:29.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724907+0000", 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025779699999999998 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724924+0000", 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7110000000000001e-05 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724934+0000", 2024-04-20T02:51:29.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5370000000000003e-06 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724984+0000", 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0791000000000002e-05 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.725011+0000", 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6299e-05 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771009+0000", 2024-04-20T02:51:29.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045998144999999997 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771050+0000", 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1727999999999998e-05 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.238057300000001, 2024-04-20T02:51:29.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058137664999999998, 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967112+0000", 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981895+0000", 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014783522 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981906+0000", 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0560000000000001e-05 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981912+0000", 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981956+0000", 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3455000000000003e-05 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981979+0000", 2024-04-20T02:51:29.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3475000000000001e-05 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025222+0000", 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043242622000000001 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025250+0000", 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8520000000000001e-05 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.815988502000003, 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045096171999999997, 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401391+0000", 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012209697 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401410+0000", 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8754e-05 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401422+0000", 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2089999999999999e-05 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.403142+0000", 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0017199100000000001 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.403157+0000", 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.526e-05 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434233+0000", 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031076289999999999 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434278+0000", 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4478999999999999e-05 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.278 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.793049312000001, 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:51:29.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.787044524000002, 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:29.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.287 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:130 3.0 3:1b476ba0:::474a8d6c2b8981e0c13c883e0d254815cfd155e523c9b654ff4e5007d0a1fc09:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.077703517, 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045176642000000003, 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128171+0000", 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070576600000000003 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128183+0000", 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1542999999999999e-05 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128191+0000", 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5839999999999995e-06 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134592+0000", 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006400233 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134784+0000", 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000191831 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135041+0000", 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000257123 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162360+0000", 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027319774000000002 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162476+0000", 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115505 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172550+0000", 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01007366 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172598+0000", 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8216000000000001e-05 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172643+0000", 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5305000000000001e-05 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:135 3.0 3:12659911:::ad802ca609ac9a6b9a438bdacacdb4fb27f17cd228f3ff5506eb31f643d749ff:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:29.293 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.072675561, 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045928208999999998, 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132493+0000", 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133291+0000", 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079715799999999996 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133301+0000", 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0533000000000001e-05 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133309+0000", 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9910000000000002e-06 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151891+0000", 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018582155 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152003+0000", 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112064 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152211+0000", 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020773100000000001 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162492+0000", 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010281047999999999 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169484+0000", 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006991929 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178312+0000", 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0088282789999999996 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178374+0000", 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2123000000000001e-05 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178423+0000", 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201999999999997e-05 2024-04-20T02:51:29.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:138 3.0 3:2a051c65:::f4051f504bf0c69c0b0fcc37af36dc6836e32eef2351291df998149766aaa9b9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.071854144, 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045257957000000001, 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:29.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133315+0000", 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133606+0000", 2024-04-20T02:51:29.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029134199999999999 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133615+0000", 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2479999999999996e-06 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133621+0000", 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7739999999999999e-06 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159641+0000", 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026020008000000001 2024-04-20T02:51:29.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159813+0000", 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017143200000000001 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160027+0000", 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021374400000000001 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.163217+0000", 2024-04-20T02:51:29.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031905869999999999 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169517+0000", 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063002290000000001 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178484+0000", 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0089668070000000002 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178534+0000", 2024-04-20T02:51:29.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9654000000000003e-05 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178574+0000", 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9929999999999999e-05 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:142 3.0 3:0f44ff3d:::ebd96dbdd0a43c00198785b2cc319b661a1b3295172ded27fbd66cbcc74da6e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.133627+0000", 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.071543146, 2024-04-20T02:51:29.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045069357999999997, 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133627+0000", 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133627+0000", 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133626+0000", 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133670+0000", 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4129999999999999e-05 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133677+0000", 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8460000000000002e-06 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133683+0000", 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9379999999999997e-06 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162538+0000", 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028855285000000001 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162659+0000", 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000120943 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162802+0000", 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014214100000000001 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.163523+0000", 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072146799999999996 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.170143+0000", 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066200490000000002 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178613+0000", 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0084695819999999998 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178658+0000", 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.579e-05 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178696+0000", 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7768000000000002e-05 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.309 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 15.116536394000001, 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049605908999999997, 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088632+0000", 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090096+0000", 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001463675 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090112+0000", 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5557999999999999e-05 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090125+0000", 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3794999999999999e-05 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090196+0000", 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0537000000000006e-05 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090224+0000", 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8555999999999999e-05 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138207+0000", 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047982536999999999 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138240+0000", 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2480000000000001e-05 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.313 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.314 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.314 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:29.314 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:29.314 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:29.315 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:29.206+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:29.315 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:29.316 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:29.316+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:29.318 INFO:tasks.rados.rados.0.smithi105.stdout:1095: finishing rollback tid 0 to smithi10548823-40 2024-04-20T02:51:29.318 INFO:tasks.rados.rados.0.smithi105.stdout:1095: finishing rollback tid 1 to smithi10548823-40 2024-04-20T02:51:29.318 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 168 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-04-20T02:51:29.318 INFO:tasks.rados.rados.0.smithi105.stdout:1093: finishing rollback tid 0 to smithi10548823-49 2024-04-20T02:51:29.318 INFO:tasks.rados.rados.0.smithi105.stdout:1093: finishing rollback tid 1 to smithi10548823-49 2024-04-20T02:51:29.318 INFO:tasks.rados.rados.0.smithi105.stdout:1093: finishing rollback tid 2 to smithi10548823-49 2024-04-20T02:51:29.318 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 96 (ObjNum 152 snap 10 seq_num 152) dirty exists 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1094: got (0) Success 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 88 (ObjNum 155 snap 10 seq_num 155) dirty exists 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1093: done (8 left) 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1094: done (7 left) 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1095: done (6 left) 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1096: done (5 left) 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1102: set_chunk oid 38 target oid 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1102: set_chunk oid 38 offset: 164864 length: 1941504 target oid smithi10548823- offset: 164864 snap 9 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1103: read oid 30 snap -1 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1103: expect (ObjNum 185 snap 17 seq_num 185) 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1104: read oid 7 snap 9 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1104: expect (ObjNum 105 snap 5 seq_num 105) 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1105: read oid 32 snap -1 2024-04-20T02:51:29.319 INFO:tasks.rados.rados.0.smithi105.stdout:1105: expect (ObjNum 183 snap 17 seq_num 183) 2024-04-20T02:51:29.320 INFO:tasks.rados.rados.0.smithi105.stdout:1106: read oid 17 snap -1 2024-04-20T02:51:29.320 INFO:tasks.rados.rados.0.smithi105.stdout:1106: expect (ObjNum 123 snap 7 seq_num 123) 2024-04-20T02:51:29.320 INFO:tasks.rados.rados.0.smithi105.stdout:1107: write oid 36 current snap is 18 2024-04-20T02:51:29.320 INFO:tasks.rados.rados.0.smithi105.stdout:1098: expect (ObjNum 170 snap 14 seq_num 170) 2024-04-20T02:51:29.383 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:29.419 INFO:tasks.rados.rados.0.smithi105.stdout:1097: finishing set_chunk to oid 1 2024-04-20T02:51:29.420 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 108 (ObjNum 133 snap 9 seq_num 133) dirty exists 2024-04-20T02:51:29.420 INFO:tasks.rados.rados.0.smithi105.stdout:1099: expect (ObjNum 165 snap 14 seq_num 165) 2024-04-20T02:51:29.420 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:29.420+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:29.451 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:29.451 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:29.462 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:29.533 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:29.534 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:29.533+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:29.554 INFO:tasks.rados.rados.0.smithi105.stdout:1101: got (0) Success 2024-04-20T02:51:29.554 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 55 (ObjNum 103 snap 5 seq_num 103) dirty exists 2024-04-20T02:51:29.554 INFO:tasks.rados.rados.0.smithi105.stdout:1102: finishing set_chunk to oid 38 2024-04-20T02:51:29.554 INFO:tasks.rados.rados.0.smithi105.stdout:1102: got expected ERANGE (src dne) 2024-04-20T02:51:29.554 INFO:tasks.rados.rados.0.smithi105.stdout:1104: expect (ObjNum 105 snap 5 seq_num 105) 2024-04-20T02:51:29.618 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:29.618+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:29.633 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:29.633 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:29.633 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:29.633 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:29.643 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:29.716 INFO:tasks.rados.rados.0.smithi105.stdout:1107: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-04-20T02:51:29.719 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:29.719+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:29.724 INFO:tasks.rados.rados.0.smithi105.stdout:1107: writing smithi10548823-36 from 793330 to 1483112 tid 1 2024-04-20T02:51:29.730 INFO:tasks.rados.rados.0.smithi105.stdout:1107: writing smithi10548823-36 from 2035461 to 2653192 tid 2 2024-04-20T02:51:29.734 INFO:tasks.rados.rados.0.smithi105.stdout:1107: writing smithi10548823-36 from 3101545 to 3193330 tid 3 2024-04-20T02:51:29.734 INFO:tasks.rados.rados.0.smithi105.stdout:1097: done (10 left) 2024-04-20T02:51:29.734 INFO:tasks.rados.rados.0.smithi105.stdout:1098: done (9 left) 2024-04-20T02:51:29.734 INFO:tasks.rados.rados.0.smithi105.stdout:1099: done (8 left) 2024-04-20T02:51:29.734 INFO:tasks.rados.rados.0.smithi105.stdout:1101: done (7 left) 2024-04-20T02:51:29.734 INFO:tasks.rados.rados.0.smithi105.stdout:1102: done (6 left) 2024-04-20T02:51:29.734 INFO:tasks.rados.rados.0.smithi105.stdout:1104: done (5 left) 2024-04-20T02:51:29.735 INFO:tasks.rados.rados.0.smithi105.stdout:1108: read oid 21 snap -1 2024-04-20T02:51:29.735 INFO:tasks.rados.rados.0.smithi105.stdout:1108: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-20T02:51:29.735 INFO:tasks.rados.rados.0.smithi105.stdout:1109: read oid 27 snap 11 2024-04-20T02:51:29.735 INFO:tasks.rados.rados.0.smithi105.stdout:1109: expect (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:51:29.735 INFO:tasks.rados.rados.0.smithi105.stdout:1110: snap_create 2024-04-20T02:51:29.738 INFO:tasks.rados.rados.0.smithi105.stdout:1108: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-20T02:51:29.774 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:29.797 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:29.817 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:29.819 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:29.820+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:29.831 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:29.832 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:29.833 INFO:tasks.rados.rados.0.smithi105.stdout:1109: expect (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:51:29.837 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:29.843 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:29.858 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:29.878 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:29.898 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:29.900 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:29.900 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:29.900 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:29.900 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:29.900 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:29.900 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:29.900 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:29.900 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:29.921 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:29.921+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:29.988 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:29.989 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:29.999 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:30.043 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:30.043+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:1107: finishing write tid 1 to smithi10548823-36 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:1107: finishing write tid 2 to smithi10548823-36 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:1107: finishing write tid 3 to smithi10548823-36 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:1107: finishing write tid 4 to smithi10548823-36 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:1107: finishing write tid 5 to smithi10548823-36 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:1107: oid 36 updating version 0 to 172 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:1107: oid 36 updating version 172 to 173 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:1107: oid 36 version 173 is already newer than 170 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:1107: oid 36 version 173 is already newer than 171 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 173 (ObjNum 190 snap 18 seq_num 190) dirty exists 2024-04-20T02:51:30.044 INFO:tasks.rados.rados.0.smithi105.stdout:1107: left oid 36 (ObjNum 190 snap 18 seq_num 190) 2024-04-20T02:51:30.123 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:30.123+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:30.126 INFO:tasks.rados.rados.0.smithi105.stdout:1107: done (7 left) 2024-04-20T02:51:30.127 INFO:tasks.rados.rados.0.smithi105.stdout:1108: done (6 left) 2024-04-20T02:51:30.127 INFO:tasks.rados.rados.0.smithi105.stdout:1109: done (5 left) 2024-04-20T02:51:30.127 INFO:tasks.rados.rados.0.smithi105.stdout:1110: done (4 left) 2024-04-20T02:51:30.127 INFO:tasks.rados.rados.0.smithi105.stdout:1111: write oid 4 current snap is 19 2024-04-20T02:51:30.127 INFO:tasks.rados.rados.0.smithi105.stdout:1111: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-04-20T02:51:30.133 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:30.134 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:30.134 INFO:tasks.rados.rados.0.smithi105.stdout:1111: writing smithi10548823-4 from 410137 to 891521 tid 1 2024-04-20T02:51:30.140 INFO:tasks.rados.rados.0.smithi105.stdout:1111: writing smithi10548823-4 from 1410296 to 2171685 tid 2 2024-04-20T02:51:30.146 INFO:tasks.rados.rados.0.smithi105.stdout:1111: writing smithi10548823-4 from 2728968 to 3210137 tid 3 2024-04-20T02:51:30.150 INFO:tasks.rados.rados.0.smithi105.stdout:1112: write (excl) oid 46 current snap is 19 2024-04-20T02:51:30.150 INFO:tasks.rados.rados.0.smithi105.stdout:1112: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-04-20T02:51:30.153 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:30.153 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:51:30.157 INFO:tasks.rados.rados.0.smithi105.stdout:1112: writing smithi10548823-46 from 426944 to 983577 tid 1 2024-04-20T02:51:30.179 INFO:tasks.rados.rados.0.smithi105.stdout:1112: writing smithi10548823-46 from 1468778 to 2090178 tid 2 2024-04-20T02:51:30.179 INFO:tasks.rados.rados.0.smithi105.stdout:1112: writing smithi10548823-46 from 2872744 to 3226944 tid 3 2024-04-20T02:51:30.179 INFO:tasks.rados.rados.0.smithi105.stdout:1113: read oid 34 snap -1 2024-04-20T02:51:30.179 INFO:tasks.rados.rados.0.smithi105.stdout:1113: expect (ObjNum 170 snap 14 seq_num 170) 2024-04-20T02:51:30.179 INFO:tasks.rados.rados.0.smithi105.stdout:1114: tier_evict oid 28 2024-04-20T02:51:30.179 INFO:tasks.rados.rados.0.smithi105.stdout:1114: tier_evict oid 28 snap -1 2024-04-20T02:51:30.180 INFO:tasks.rados.rados.0.smithi105.stdout:1115: write (excl) oid 20 current snap is 19 2024-04-20T02:51:30.180 INFO:tasks.rados.rados.0.smithi105.stdout:1115: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-04-20T02:51:30.180 INFO:tasks.rados.rados.0.smithi105.stdout:1115: writing smithi10548823-20 from 443751 to 1075633 tid 1 2024-04-20T02:51:30.187 INFO:tasks.rados.rados.0.smithi105.stdout:1115: writing smithi10548823-20 from 1810907 to 2575965 tid 2 2024-04-20T02:51:30.198 INFO:tasks.rados.rados.0.smithi105.stdout:1115: writing smithi10548823-20 from 3067461 to 3243751 tid 3 2024-04-20T02:51:30.198 INFO:tasks.rados.rados.0.smithi105.stdout:1116: read oid 24 snap -1 2024-04-20T02:51:30.198 INFO:tasks.rados.rados.0.smithi105.stdout:1116: expect (ObjNum 180 snap 15 seq_num 180) 2024-04-20T02:51:30.198 INFO:tasks.rados.rados.0.smithi105.stdout:1117: tier_promote oid 2 2024-04-20T02:51:30.199 INFO:tasks.rados.rados.0.smithi105.stdout:1118: write oid 6 current snap is 19 2024-04-20T02:51:30.199 INFO:tasks.rados.rados.0.smithi105.stdout:1118: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-04-20T02:51:30.201 INFO:tasks.rados.rados.0.smithi105.stdout:1118: writing smithi10548823-6 from 460558 to 1167689 tid 1 2024-04-20T02:51:30.209 INFO:tasks.rados.rados.0.smithi105.stdout:1118: writing smithi10548823-6 from 1869389 to 2378105 tid 2 2024-04-20T02:51:30.214 INFO:tasks.rados.rados.0.smithi105.stdout:1118: writing smithi10548823-6 from 3094884 to 3260558 tid 3 2024-04-20T02:51:30.214 INFO:tasks.rados.rados.0.smithi105.stdout:1119: tier_promote oid 18 2024-04-20T02:51:30.214 INFO:tasks.rados.rados.0.smithi105.stdout:1120: set_chunk oid 22 target oid 2024-04-20T02:51:30.223 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:30.224+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:30.299 INFO:tasks.rados.rados.0.smithi105.stdout:1120: set_chunk oid 22 offset: 325632 length: 482304 target oid a5f3fe64939001858437bf009a79a5587d299b5a278d9e2e9967813883029964 offset: 0 snap -1 2024-04-20T02:51:30.300 INFO:tasks.rados.rados.0.smithi105.stdout:1121: read oid 13 snap -1 2024-04-20T02:51:30.300 INFO:tasks.rados.rados.0.smithi105.stdout:1121: expect (ObjNum 151 snap 10 seq_num 151) 2024-04-20T02:51:30.300 INFO:tasks.rados.rados.0.smithi105.stdout:1122: write oid 48 current snap is 19 2024-04-20T02:51:30.300 INFO:tasks.rados.rados.0.smithi105.stdout:1122: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-04-20T02:51:30.306 INFO:tasks.rados.rados.0.smithi105.stdout:1122: writing smithi10548823-48 from 477365 to 1259745 tid 1 2024-04-20T02:51:30.316 INFO:tasks.rados.rados.0.smithi105.stdout:1122: writing smithi10548823-48 from 1927871 to 2696598 tid 2 2024-04-20T02:51:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1122: writing smithi10548823-48 from 3122307 to 3277365 tid 3 2024-04-20T02:51:30.319 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:30.320 INFO:tasks.rados.rados.0.smithi105.stdout:1105: expect (ObjNum 183 snap 17 seq_num 183) 2024-04-20T02:51:30.324 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:30.325+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:30.350 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:30.351 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:30.365 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:30.426 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:30.426+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:30.526 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:30.527+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:30.546 INFO:tasks.rados.rados.0.smithi105.stdout:1103: expect (ObjNum 185 snap 17 seq_num 185) 2024-04-20T02:51:30.548 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:30.548 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:30.559 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:30.627 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:30.628+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:30.704 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:30.704 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:30.715 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:30.756 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:30.757+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:30.773 INFO:tasks.rados.rados.0.smithi105.stdout:1106: expect (ObjNum 123 snap 7 seq_num 123) 2024-04-20T02:51:30.829 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:30.830+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:30.885 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:30.885 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:30.886 INFO:tasks.rados.rados.0.smithi105.stdout:1114: got (22) Invalid argument 2024-04-20T02:51:30.886 INFO:tasks.rados.rados.0.smithi105.stdout:1113: expect (ObjNum 170 snap 14 seq_num 170) 2024-04-20T02:51:30.895 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:30.936 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:30.937+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:30.980 INFO:tasks.rados.rados.0.smithi105.stdout:1116: expect (ObjNum 180 snap 15 seq_num 180) 2024-04-20T02:51:31.031 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:31.032+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:31.065 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:31.065 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:31.076 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:31.116 INFO:tasks.rados.rados.0.smithi105.stdout:1112: finishing write tid 1 to smithi10548823-46 2024-04-20T02:51:31.116 INFO:tasks.rados.rados.0.smithi105.stdout:1112: finishing write tid 2 to smithi10548823-46 2024-04-20T02:51:31.116 INFO:tasks.rados.rados.0.smithi105.stdout:1111: finishing write tid 1 to smithi10548823-4 2024-04-20T02:51:31.116 INFO:tasks.rados.rados.0.smithi105.stdout:1111: finishing write tid 2 to smithi10548823-4 2024-04-20T02:51:31.116 INFO:tasks.rados.rados.0.smithi105.stdout:1100: done (15 left) 2024-04-20T02:51:31.116 INFO:tasks.rados.rados.0.smithi105.stdout:1103: done (14 left) 2024-04-20T02:51:31.116 INFO:tasks.rados.rados.0.smithi105.stdout:1105: done (13 left) 2024-04-20T02:51:31.116 INFO:tasks.rados.rados.0.smithi105.stdout:1106: done (12 left) 2024-04-20T02:51:31.117 INFO:tasks.rados.rados.0.smithi105.stdout:1113: done (11 left) 2024-04-20T02:51:31.117 INFO:tasks.rados.rados.0.smithi105.stdout:1114: done (10 left) 2024-04-20T02:51:31.117 INFO:tasks.rados.rados.0.smithi105.stdout:1116: done (9 left) 2024-04-20T02:51:31.117 INFO:tasks.rados.rados.0.smithi105.stdout:1123: read oid 28 snap -1 2024-04-20T02:51:31.117 INFO:tasks.rados.rados.0.smithi105.stdout:1123: expect (ObjNum 181 snap 15 seq_num 181) 2024-04-20T02:51:31.117 INFO:tasks.rados.rados.0.smithi105.stdout:1124: write oid 33 current snap is 19 2024-04-20T02:51:31.117 INFO:tasks.rados.rados.0.smithi105.stdout:1111: finishing write tid 3 to smithi10548823-4 2024-04-20T02:51:31.117 INFO:tasks.rados.rados.0.smithi105.stdout:1124: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-04-20T02:51:31.121 INFO:tasks.rados.rados.0.smithi105.stdout:1124: writing smithi10548823-33 from 494172 to 951801 tid 1 2024-04-20T02:51:31.128 INFO:tasks.rados.rados.0.smithi105.stdout:1124: writing smithi10548823-33 from 1586353 to 2098738 tid 2 2024-04-20T02:51:31.135 INFO:tasks.rados.rados.0.smithi105.stdout:1124: writing smithi10548823-33 from 2749730 to 3294172 tid 3 2024-04-20T02:51:31.135 INFO:tasks.rados.rados.0.smithi105.stdout:1125: set_chunk oid 8 target oid 2024-04-20T02:51:31.137 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:51:31.160 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:31.161+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:31.233 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:31.233+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:31.254 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:31.255 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:31.265 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:31.334 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:31.334+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:31.434 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:31.434 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:31.435 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:31.435+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:31.444 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:31.538 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:31.539+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:31.572 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:31.572 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:31.583 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:31.639 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:31.640+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:31.711 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:31.712 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:31.723 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:31.764 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:31.765+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:31.841 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:31.841+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:31.891 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:31.892 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:31.902 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:31.913 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:31.956 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:31.957+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:32.043 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:32.043+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:32.063 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:32.074 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:32.074 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:32.082 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:32.084 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:32.101 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:32.123 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:32.144 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:32.144+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:32.144 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:51:32.144 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:51:32.145 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:51:32.145 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:32.150 INFO:tasks.rados.rados.0.smithi105.stdout:1125: set_chunk oid 8 offset: 809984 length: 833024 target oid d7f933489e6e6b4cf801c3fb2c5aa7adbce03d1fb4fbc690db75afcecc89e93a offset: 0 snap -1 2024-04-20T02:51:32.150 INFO:tasks.rados.rados.0.smithi105.stdout:1126: snap_remove snap 16 2024-04-20T02:51:32.168 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:32.188 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:32.208 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:32.209 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:32.226 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:32.226 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:32.226 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:32.226 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:32.226 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:32.236 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:32.277 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:32.278+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:32.345 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:32.345+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:32.407 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:32.407 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:32.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:32.407 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:32.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:152 3.2 e48/47 3:54f0c00e:::2528189ec1c891f179c539fa23af2c87b9e8d2c19f70e70fac8934d3cc51ccee:head v 48'96, mlcod=0'0)", 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 26.973980679, 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020196797999999998, 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433091+0000", 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000185589 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433102+0000", 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433108+0000", 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5660000000000003e-06 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439137+0000", 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060284329999999997 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439166+0000", 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9278e-05 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453075+0000", 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013909163 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453103+0000", 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7725999999999998e-05 2024-04-20T02:51:32.411 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:154 3.2 e48/47 3:7d72d9ce:::7b4ada8765f224bb4a516820a83f51a402b4ee671c5f45bb3fbe5e6656cf1f14:head v 48'97, mlcod=42'91)", 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 26.970075047000002, 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016320239, 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:32.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436812+0000", 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.436811+0000", 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437600+0000", 2024-04-20T02:51:32.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078939699999999995 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437613+0000", 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2588e-05 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.437624+0000", 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0542e-05 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442541+0000", 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049178809999999998 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.442565+0000", 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3558e-05 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453110+0000", 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010545172 2024-04-20T02:51:32.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453132+0000", 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1929e-05 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:155 3.2 e48/47 3:715fab8c:::b945a2927747be608263bdbe1fe814f665c73382cb3375555da5aa95a92567ff:head v 48'99, mlcod=42'91)", 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 26.967287991999999, 2024-04-20T02:51:32.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017321785999999999, 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439599+0000", 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439598+0000", 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439954+0000", 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00035586299999999999 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439970+0000", 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5889e-05 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439981+0000", 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0754000000000001e-05 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443643+0000", 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036621259999999999 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.443671+0000", 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8289e-05 2024-04-20T02:51:32.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456885+0000", 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013213964999999999 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.456921+0000", 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.420 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1635 3.2 e51/49 3:67667c16:::7aea211498f946c134ed12b1f5632511a2f083a0eb2114d1051f38dc7326b113:head v 51'100, mlcod=0'0)", 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.559397045000001, 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019392829, 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847489+0000", 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850819+0000", 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033292069999999998 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850831+0000", 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2320000000000001e-05 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850838+0000", 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2980000000000003e-06 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850874+0000", 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5800999999999998e-05 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850890+0000", 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5536000000000002e-05 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866849+0000", 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015959346999999999 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866883+0000", 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3776999999999999e-05 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.958248806, 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:32.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1665 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'78, mlcod=50'73)", 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.883648953000002, 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025918594, 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523236+0000", 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523658+0000", 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000422082 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523676+0000", 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7716000000000001e-05 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523686+0000", 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9790000000000005e-06 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523734+0000", 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7579e-05 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523759+0000", 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5593e-05 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549122+0000", 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025363087999999999 2024-04-20T02:51:32.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549157+0000", 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4242999999999997e-05 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 21.951740573999999, 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034212370999999998, 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.434 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:32.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456260+0000", 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011140480000000001 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456273+0000", 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456280+0000", 2024-04-20T02:51:32.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1130000000000003e-06 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456331+0000", 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1446999999999997e-05 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456357+0000", 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6001999999999999e-05 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489309+0000", 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032951972000000003 2024-04-20T02:51:32.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489359+0000", 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9552000000000003e-05 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1775 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'41, mlcod=53'38)", 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.653359084000002, 2024-04-20T02:51:32.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038672274999999999, 2024-04-20T02:51:32.439 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.439 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.439 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:32.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:32.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:32.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:32.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754868+0000", 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013404619999999999 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754881+0000", 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.323e-05 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754892+0000", 2024-04-20T02:51:32.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1154e-05 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754967+0000", 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4326999999999995e-05 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.755006+0000", 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9338000000000002e-05 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792154+0000", 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037147924999999998 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792200+0000", 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6044999999999997e-05 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1776 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'42, mlcod=53'38)", 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.633378906000001, 2024-04-20T02:51:32.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019186657999999999, 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773507+0000", 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774271+0000", 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076432799999999999 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774284+0000", 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2483000000000001e-05 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774294+0000", 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6569999999999992e-06 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774338+0000", 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4576000000000001e-05 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774358+0000", 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9449999999999998e-05 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792664+0000", 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018306413000000001 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792695+0000", 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0685000000000002e-05 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1822 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'94, mlcod=55'92)", 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.303336950999999, 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031823907999999998, 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:32.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:32.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103786+0000", 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000236658 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103797+0000", 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0951000000000001e-05 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103804+0000", 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0960000000000003e-06 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103860+0000", 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5781e-05 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103887+0000", 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6999999999999999e-05 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135333+0000", 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031446004 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135374+0000", 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0778999999999997e-05 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1823 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'95, mlcod=55'92)", 2024-04-20T02:51:32.453 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.288849944999999, 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017980995999999999, 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118036+0000", 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118363+0000", 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00032734600000000001 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118381+0000", 2024-04-20T02:51:32.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7816e-05 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118391+0000", 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0155e-05 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119591+0000", 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011995560000000001 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119618+0000", 2024-04-20T02:51:32.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7475e-05 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135988+0000", 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016370096000000001 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.136018+0000", 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9745999999999999e-05 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1836 4.1 e58/46 4:8b927dd6:::smithi10548823-28:head v 58'99, mlcod=56'96)", 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 16.921320943000001, 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030374581000000001, 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485930+0000", 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00036455800000000002 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485941+0000", 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1017e-05 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485950+0000", 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.85e-06 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486000+0000", 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9589999999999998e-05 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486025+0000", 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.529e-05 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515909+0000", 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029884131000000001 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515941+0000", 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1569000000000002e-05 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.461 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.908066038999999, 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:32.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:51:32.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.90800059, 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:51:32.466 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:51:32.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.470 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.907975092000001, 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.471 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:51:32.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1891 4.17 e61/60 4:e9876a63:::smithi10548823-30:head v 61'45, mlcod=0'0)", 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:32.475 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.893429296000001, 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018777245000000001, 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513457+0000", 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514596+0000", 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011393289999999999 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514612+0000", 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6311e-05 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514623+0000", 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056999999999999e-05 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514670+0000", 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6557999999999997e-05 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514698+0000", 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8121000000000001e-05 2024-04-20T02:51:32.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.532204+0000", 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017505969999999999 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.532235+0000", 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0731000000000001e-05 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1892 4.17 e61/60 4:e9876a63:::smithi10548823-30:head v 61'46, mlcod=0'0)", 2024-04-20T02:51:32.479 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.870090312, 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020303921999999999, 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:32.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536796+0000", 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537578+0000", 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078217499999999997 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537592+0000", 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3721e-05 2024-04-20T02:51:32.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537602+0000", 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0380000000000001e-05 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537638+0000", 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5559000000000003e-05 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537657+0000", 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9242e-05 2024-04-20T02:51:32.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.557064+0000", 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019406970999999999 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.557101+0000", 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6572000000000002e-05 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.483 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2004 4.13 e65/63 4:cfd1e4d4:::smithi10548823-9:head v 65'159, mlcod=64'156)", 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5587500439999999, 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036995865000000003, 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:32.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849186+0000", 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010492889999999999 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849197+0000", 2024-04-20T02:51:32.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1365999999999999e-05 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849204+0000", 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1069999999999999e-06 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849250+0000", 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5213999999999998e-05 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849279+0000", 2024-04-20T02:51:32.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9149000000000002e-05 2024-04-20T02:51:32.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885093+0000", 2024-04-20T02:51:32.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035813921999999998 2024-04-20T02:51:32.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885133+0000", 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0077999999999998e-05 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2005 4.13 e65/63 4:cfd1e4d4:::smithi10548823-9:head v 65'160, mlcod=64'156)", 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.5388890770000003, 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.01736478, 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867997+0000", 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868547+0000", 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000550287 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868563+0000", 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5676e-05 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868572+0000", 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0440000000000001e-06 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.874844+0000", 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0062724460000000001 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.874865+0000", 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0712000000000001e-05 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885334+0000", 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010468691 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885363+0000", 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8923000000000001e-05 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2029 3.2 e66/49 3:70b121b2:::b13b8a8d7724a2b6951a53e14e5e568e1aa45f9c18b0373ab4a92ed71f345848:head v 66'106, mlcod=53'104)", 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:24.119464+0000", 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.28742336, 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023218222, 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.492 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.119464+0000", 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.119464+0000", 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.119462+0000", 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:32.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122123+0000", 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002661629 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122140+0000", 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6290000000000002e-05 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122147+0000", 2024-04-20T02:51:32.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.695e-06 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122201+0000", 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.393e-05 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122228+0000", 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6367999999999999e-05 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.142651+0000", 2024-04-20T02:51:32.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020423884 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.142682+0000", 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0301999999999999e-05 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:32.496 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:32.499 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:32.498+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:32.500 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:32.501 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:51:32.502 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:51:32.502 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-04-20T02:51:32.117780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334233857,"num_objects":251,"num_object_clones":63,"num_object_copies":753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":6,"num_read":2004,"num_read_kb":1680870,"num_write":1407,"num_write_kb":387388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":70676298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":64,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":867620,"kb_used_data":629484,"kb_used_omap":12,"kb_used_meta":237235,"kb_avail":754107100,"statfs":{"total":773094113280,"available":772205670400,"internally_reserved":0,"allocated":644591616,"data_stored":759194091,"data_compressed":15068766,"data_compressed_allocated":15937536,"data_compressed_original":36895490,"omap_allocated":12807,"internal_metadata":242929145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5109998,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-24,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":101,"num_read_kb":117314,"num_write":19,"num_write_kb":5758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.996978"},"pg_stats":[{"pgid":"4.19","version":"69'165","reported_seq":384,"reported_epoch":72,"state":"remapped+peering","last_fresh":"2024-04-20T02:51:29.131643+0000","last_change":"2024-04-20T02:51:29.131643+0000","last_active":"2024-04-20T02:51:29.131643+0000","last_peered":"2024-04-20T02:51:24.573495+0000","last_clean":"2024-04-20T02:51:24.573495+0000","last_became_active":"2024-04-20T02:51:23.563438+0000","last_became_peered":"2024-04-20T02:51:23.563438+0000","last_unstale":"2024-04-20T02:51:29.131643+0000","last_undegraded":"2024-04-20T02:51:29.131643+0000","last_fullsized":"2024-04-20T02:51:29.131643+0000","mapping_epoch":72,"log_start":"67'162","ondisk_log_start":"67'162","created":60,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":159,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012961120000000001,"stat_sum":{"num_bytes":18294509,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":108,"num_read_kb":104733,"num_write":60,"num_write_kb":15897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7389695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.18","version":"64'49","reported_seq":186,"reported_epoch":72,"state":"remapped+peering","last_fresh":"2024-04-20T02:51:29.131274+0000","last_change":"2024-04-20T02:51:29.131437+0000","last_active":"2024-04-20T02:51:28.125678+0000","last_peered":"2024-04-20T02:51:27.738349+0000","last_clean":"2024-04-20T02:50:58.916741+0000","last_became_active":"2024-04-20T02:51:21.510024+0000","last_became_peered":"2024-04-20T02:51:21.510024+0000","last_unstale":"2024-04-20T02:51:29.131274+0000","last_undegraded":"2024-04-20T02:51:29.131274+0000","last_fullsized":"2024-04-20T02:51:29.131274+0000","mapping_epoch":72,"log_start":"42'46","ondisk_log_start":"42'46","created":60,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":46,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5072097,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":75484,"num_write":20,"num_write_kb":4675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6,7],"up_primary":5,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"61'47","reported_seq":222,"reported_epoch":66,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:23.556973+0000","last_change":"2024-04-20T02:51:21.564054+0000","last_active":"2024-04-20T02:51:23.556973+0000","last_peered":"2024-04-20T02:51:23.556973+0000","last_clean":"2024-04-20T02:51:18.558395+0000","last_became_active":"2024-04-20T02:51:21.512817+0000","last_became_peered":"2024-04-20T02:51:21.512817+0000","last_unstale":"2024-04-20T02:51:23.556973+0000","last_undegraded":"2024-04-20T02:51:23.556973+0000","last_fullsized":"2024-04-20T02:51:23.556973+0000","mapping_epoch":63,"log_start":"56'43","ondisk_log_start":"56'43","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":40,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6373913,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":59515,"num_write":23,"num_write_kb":6860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2757906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[0,1,5],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.1","version":"58'101","reported_seq":296,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-20T02:51:28.126021+0000","last_change":"2024-04-20T02:51:24.576137+0000","last_active":"2024-04-20T02:51:28.126021+0000","last_peered":"2024-04-20T02:51:28.126021+0000","last_clean":"2024-04-20T02:51:28.126021+0000","last_became_active":"2024-04-20T02:51:18.508043+0000","last_became_peered":"2024-04-20T02:51:18.508043+0000","last_unstale":"2024-04-20T02:51:28.126021+0000","last_undegraded":"2024-04-20T02:51:28.126021+0000","last_fullsized":"2024-04-20T02:51:28.126021+0000","mapping_epoch":46,"log_start":"57'97","ondisk_log_start":"57'97","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":97,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023371690000000001,"stat_sum":{"num_bytes":9042365,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":63,"num_read_kb":63720,"num_write":41,"num_write_kb":11120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3703652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.117620+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:51:26.117620+0000","last_peered":"2024-04-20T02:51:26.117620+0000","last_clean":"2024-04-20T02:51:26.117620+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:51:26.117620+0000","last_undegraded":"2024-04-20T02:51:26.117620+0000","last_fullsized":"2024-04-20T02:51:26.117620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.432280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"67'74","reported_seq":256,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.117709+0000","last_change":"2024-04-20T02:51:24.572938+0000","last_active":"2024-04-20T02:51:26.117709+0000","last_peered":"2024-04-20T02:51:26.117709+0000","last_clean":"2024-04-20T02:51:26.117709+0000","last_became_active":"2024-04-20T02:51:18.505708+0000","last_became_peered":"2024-04-20T02:51:18.505708+0000","last_unstale":"2024-04-20T02:51:26.117709+0000","last_undegraded":"2024-04-20T02:51:26.117709+0000","last_fullsized":"2024-04-20T02:51:26.117709+0000","mapping_epoch":22,"log_start":"64'70","ondisk_log_start":"64'70","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":66,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.922550+0000","scrub_duration":0.001820667,"objects_trimmed":0,"snaptrim_duration":0.00092831499999999998,"stat_sum":{"num_bytes":9040558,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.148955+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:51:26.148955+0000","last_peered":"2024-04-20T02:51:26.148955+0000","last_clean":"2024-04-20T02:51:26.148955+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:51:26.148955+0000","last_undegraded":"2024-04-20T02:51:26.148955+0000","last_fullsized":"2024-04-20T02:51:26.148955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":62,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-20T02:51:28.128948+0000","last_change":"2024-04-20T02:50:28.091938+0000","last_active":"2024-04-20T02:51:28.128948+0000","last_peered":"2024-04-20T02:51:28.128948+0000","last_clean":"2024-04-20T02:51:28.128948+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:51:28.128948+0000","last_undegraded":"2024-04-20T02:51:28.128948+0000","last_fullsized":"2024-04-20T02:51:28.128948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:34.990718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"64'161","reported_seq":405,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-20T02:51:29.320388+0000","last_change":"2024-04-20T02:51:24.598492+0000","last_active":"2024-04-20T02:51:29.320388+0000","last_peered":"2024-04-20T02:51:29.320388+0000","last_clean":"2024-04-20T02:51:29.320388+0000","last_became_active":"2024-04-20T02:51:18.502004+0000","last_became_peered":"2024-04-20T02:51:18.502004+0000","last_unstale":"2024-04-20T02:51:29.320388+0000","last_undegraded":"2024-04-20T02:51:29.320388+0000","last_fullsized":"2024-04-20T02:51:29.320388+0000","mapping_epoch":47,"log_start":"61'157","ondisk_log_start":"61'157","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'157","last_scrub_stamp":"2024-04-20T02:51:19.377186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.377186+0000","objects_scrubbed":5,"log_size":4,"log_dups_size":153,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:38.408177+0000","scrub_duration":0.0026503899999999999,"objects_trimmed":0,"snaptrim_duration":0.011181791999999999,"stat_sum":{"num_bytes":8320028,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":52029,"num_write":4,"num_write_kb":1068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"3.1","version":"72'129","reported_seq":209,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-20T02:51:29.161246+0000","last_change":"2024-04-20T02:50:34.239216+0000","last_active":"2024-04-20T02:51:29.161246+0000","last_peered":"2024-04-20T02:51:29.161246+0000","last_clean":"2024-04-20T02:51:29.161246+0000","last_became_active":"2024-04-20T02:50:34.238906+0000","last_became_peered":"2024-04-20T02:50:34.238906+0000","last_unstale":"2024-04-20T02:51:29.161246+0000","last_undegraded":"2024-04-20T02:51:29.161246+0000","last_fullsized":"2024-04-20T02:51:29.161246+0000","mapping_epoch":21,"log_start":"65'126","ondisk_log_start":"65'126","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":151,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.390119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34749029,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":120,"num_read_kb":6401,"num_write":156,"num_write_kb":53865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":73,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-20T02:51:28.222310+0000","last_change":"2024-04-20T02:51:28.222310+0000","last_active":"2024-04-20T02:51:28.222310+0000","last_peered":"2024-04-20T02:51:28.222310+0000","last_clean":"2024-04-20T02:51:28.222310+0000","last_became_active":"2024-04-20T02:50:28.091570+0000","last_became_peered":"2024-04-20T02:50:28.091570+0000","last_unstale":"2024-04-20T02:51:28.222310+0000","last_undegraded":"2024-04-20T02:51:28.222310+0000","last_fullsized":"2024-04-20T02:51:28.222310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:28.222203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:28.222203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:39.650022+0000","scrub_duration":0.0013437259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"70'67","reported_seq":236,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-20T02:51:28.126235+0000","last_change":"2024-04-20T02:51:24.596959+0000","last_active":"2024-04-20T02:51:28.126235+0000","last_peered":"2024-04-20T02:51:28.126235+0000","last_clean":"2024-04-20T02:51:28.126235+0000","last_became_active":"2024-04-20T02:51:18.504889+0000","last_became_peered":"2024-04-20T02:51:18.504889+0000","last_unstale":"2024-04-20T02:51:28.126235+0000","last_undegraded":"2024-04-20T02:51:28.126235+0000","last_fullsized":"2024-04-20T02:51:28.126235+0000","mapping_epoch":22,"log_start":"67'63","ondisk_log_start":"67'63","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'57","last_scrub_stamp":"2024-04-20T02:51:21.386796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:21.386796+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":62,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:27.553181+0000","scrub_duration":0.0018878359999999999,"objects_trimmed":0,"snaptrim_duration":0.0099481110000000008,"stat_sum":{"num_bytes":7633807,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":11429,"num_write":6,"num_write_kb":1127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"3.2","version":"69'108","reported_seq":266,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:27.101793+0000","last_change":"2024-04-20T02:51:07.118974+0000","last_active":"2024-04-20T02:51:27.101793+0000","last_peered":"2024-04-20T02:51:27.101793+0000","last_clean":"2024-04-20T02:51:27.101793+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:51:27.101793+0000","last_undegraded":"2024-04-20T02:51:27.101793+0000","last_fullsized":"2024-04-20T02:51:27.101793+0000","mapping_epoch":49,"log_start":"53'105","ondisk_log_start":"53'105","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":113,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:42.672805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29567001,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":96,"num_read_kb":7581,"num_write":125,"num_write_kb":52895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":215,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-20T02:51:28.132952+0000","last_change":"2024-04-20T02:50:19.923431+0000","last_active":"2024-04-20T02:51:28.132952+0000","last_peered":"2024-04-20T02:51:28.132952+0000","last_clean":"2024-04-20T02:51:28.132952+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:51:28.132952+0000","last_undegraded":"2024-04-20T02:51:28.132952+0000","last_fullsized":"2024-04-20T02:51:28.132952+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:50:17.569030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:37.655607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":38,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-20T02:51:27.766321+0000","last_change":"2024-04-20T02:51:01.151894+0000","last_active":"2024-04-20T02:51:27.766321+0000","last_peered":"2024-04-20T02:51:27.766321+0000","last_clean":"2024-04-20T02:51:27.766321+0000","last_became_active":"2024-04-20T02:51:01.151544+0000","last_became_peered":"2024-04-20T02:51:01.151544+0000","last_unstale":"2024-04-20T02:51:27.766321+0000","last_undegraded":"2024-04-20T02:51:27.766321+0000","last_fullsized":"2024-04-20T02:51:27.766321+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:29.245779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"56'33","reported_seq":168,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.117629+0000","last_change":"2024-04-20T02:51:24.570125+0000","last_active":"2024-04-20T02:51:26.117629+0000","last_peered":"2024-04-20T02:51:26.117629+0000","last_clean":"2024-04-20T02:51:26.117629+0000","last_became_active":"2024-04-20T02:51:18.502179+0000","last_became_peered":"2024-04-20T02:51:18.502179+0000","last_unstale":"2024-04-20T02:51:26.117629+0000","last_undegraded":"2024-04-20T02:51:26.117629+0000","last_fullsized":"2024-04-20T02:51:26.117629+0000","mapping_epoch":22,"log_start":"53'30","ondisk_log_start":"53'30","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026058899999999998,"stat_sum":{"num_bytes":4739995,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":44474,"num_write":11,"num_write_kb":3858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"3.0","version":"59'97","reported_seq":163,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.148275+0000","last_change":"2024-04-20T02:50:34.241909+0000","last_active":"2024-04-20T02:51:26.148275+0000","last_peered":"2024-04-20T02:51:26.148275+0000","last_clean":"2024-04-20T02:51:26.148275+0000","last_became_active":"2024-04-20T02:50:34.241613+0000","last_became_peered":"2024-04-20T02:50:34.241613+0000","last_unstale":"2024-04-20T02:51:26.148275+0000","last_undegraded":"2024-04-20T02:51:26.148275+0000","last_fullsized":"2024-04-20T02:51:26.148275+0000","mapping_epoch":21,"log_start":"55'94","ondisk_log_start":"55'94","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":106,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:54.710965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23879111,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":84,"num_read_kb":3291,"num_write":126,"num_write_kb":40638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":59,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.117284+0000","last_change":"2024-04-20T02:50:28.091742+0000","last_active":"2024-04-20T02:51:26.117284+0000","last_peered":"2024-04-20T02:51:26.117284+0000","last_clean":"2024-04-20T02:51:26.117284+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:51:26.117284+0000","last_undegraded":"2024-04-20T02:51:26.117284+0000","last_fullsized":"2024-04-20T02:51:26.117284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:32.326186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"56'43","reported_seq":219,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.149072+0000","last_change":"2024-04-20T02:51:24.583519+0000","last_active":"2024-04-20T02:51:26.149072+0000","last_peered":"2024-04-20T02:51:26.149072+0000","last_clean":"2024-04-20T02:51:26.149072+0000","last_became_active":"2024-04-20T02:51:18.506025+0000","last_became_peered":"2024-04-20T02:51:18.506025+0000","last_unstale":"2024-04-20T02:51:26.149072+0000","last_undegraded":"2024-04-20T02:51:26.149072+0000","last_fullsized":"2024-04-20T02:51:26.149072+0000","mapping_epoch":46,"log_start":"55'40","ondisk_log_start":"55'40","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006798325,"stat_sum":{"num_bytes":5080805,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":42186,"num_write":19,"num_write_kb":5596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2757905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"3.3","version":"69'97","reported_seq":170,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-20T02:51:29.167314+0000","last_change":"2024-04-20T02:50:34.242061+0000","last_active":"2024-04-20T02:51:29.167314+0000","last_peered":"2024-04-20T02:51:29.167314+0000","last_clean":"2024-04-20T02:51:29.167314+0000","last_became_active":"2024-04-20T02:50:34.241638+0000","last_became_peered":"2024-04-20T02:50:34.241638+0000","last_unstale":"2024-04-20T02:51:29.167314+0000","last_undegraded":"2024-04-20T02:51:29.167314+0000","last_fullsized":"2024-04-20T02:51:29.167314+0000","mapping_epoch":21,"log_start":"56'94","ondisk_log_start":"56'94","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":100,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:58.050023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30742592,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":89,"num_read_kb":4892,"num_write":122,"num_write_kb":40344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":38,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-20T02:51:27.766350+0000","last_change":"2024-04-20T02:51:01.134074+0000","last_active":"2024-04-20T02:51:27.766350+0000","last_peered":"2024-04-20T02:51:27.766350+0000","last_clean":"2024-04-20T02:51:27.766350+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:51:27.766350+0000","last_undegraded":"2024-04-20T02:51:27.766350+0000","last_fullsized":"2024-04-20T02:51:27.766350+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:38.091564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"56'17","reported_seq":154,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.148775+0000","last_change":"2024-04-20T02:51:24.584535+0000","last_active":"2024-04-20T02:51:26.148775+0000","last_peered":"2024-04-20T02:51:26.148775+0000","last_clean":"2024-04-20T02:51:26.148775+0000","last_became_active":"2024-04-20T02:51:18.509276+0000","last_became_peered":"2024-04-20T02:51:18.509276+0000","last_unstale":"2024-04-20T02:51:26.148775+0000","last_undegraded":"2024-04-20T02:51:26.148775+0000","last_fullsized":"2024-04-20T02:51:26.148775+0000","mapping_epoch":22,"log_start":"53'14","ondisk_log_start":"53'14","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:51:22.420320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:22.420320+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.257627+0000","scrub_duration":0.0016858400000000001,"objects_trimmed":0,"snaptrim_duration":0.0078105229999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.2","version":"0'0","reported_seq":127,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.166518+0000","last_change":"2024-04-20T02:51:24.585106+0000","last_active":"2024-04-20T02:51:26.166518+0000","last_peered":"2024-04-20T02:51:26.166518+0000","last_clean":"2024-04-20T02:51:26.166518+0000","last_became_active":"2024-04-20T02:51:18.506462+0000","last_became_peered":"2024-04-20T02:51:18.506462+0000","last_unstale":"2024-04-20T02:51:26.166518+0000","last_undegraded":"2024-04-20T02:51:26.166518+0000","last_fullsized":"2024-04-20T02:51:26.166518+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082405940000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.148665+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:51:26.148665+0000","last_peered":"2024-04-20T02:51:26.148665+0000","last_clean":"2024-04-20T02:51:26.148665+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:51:26.148665+0000","last_undegraded":"2024-04-20T02:51:26.148665+0000","last_fullsized":"2024-04-20T02:51:26.148665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"48'47","reported_seq":185,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-04-20T02:51:27.766237+0000","last_change":"2024-04-20T02:51:24.572684+0000","last_active":"2024-04-20T02:51:27.766237+0000","last_peered":"2024-04-20T02:51:27.766237+0000","last_clean":"2024-04-20T02:51:27.766237+0000","last_became_active":"2024-04-20T02:51:20.501370+0000","last_became_peered":"2024-04-20T02:51:20.501370+0000","last_unstale":"2024-04-20T02:51:27.766237+0000","last_undegraded":"2024-04-20T02:51:27.766237+0000","last_fullsized":"2024-04-20T02:51:27.766237+0000","mapping_epoch":62,"log_start":"42'44","ondisk_log_start":"42'44","created":22,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":44,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086875600000000004,"stat_sum":{"num_bytes":5072097,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":41631,"num_write":18,"num_write_kb":4274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.9","version":"71'168","reported_seq":354,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-20T02:51:28.486343+0000","last_change":"2024-04-20T02:51:24.576236+0000","last_active":"2024-04-20T02:51:28.486343+0000","last_peered":"2024-04-20T02:51:28.486343+0000","last_clean":"2024-04-20T02:51:28.486343+0000","last_became_active":"2024-04-20T02:51:18.509871+0000","last_became_peered":"2024-04-20T02:51:18.509871+0000","last_unstale":"2024-04-20T02:51:28.486343+0000","last_undegraded":"2024-04-20T02:51:28.486343+0000","last_fullsized":"2024-04-20T02:51:28.486343+0000","mapping_epoch":22,"log_start":"71'165","ondisk_log_start":"71'165","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":165,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029055800000000001,"stat_sum":{"num_bytes":15535224,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":118,"num_read_kb":118596,"num_write":63,"num_write_kb":16725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.a","version":"69'39","reported_seq":180,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:27.103047+0000","last_change":"2024-04-20T02:51:24.569948+0000","last_active":"2024-04-20T02:51:27.103047+0000","last_peered":"2024-04-20T02:51:27.103047+0000","last_clean":"2024-04-20T02:51:27.103047+0000","last_became_active":"2024-04-20T02:50:35.413503+0000","last_became_peered":"2024-04-20T02:50:35.413503+0000","last_unstale":"2024-04-20T02:51:27.103047+0000","last_undegraded":"2024-04-20T02:51:27.103047+0000","last_fullsized":"2024-04-20T02:51:27.103047+0000","mapping_epoch":22,"log_start":"69'36","ondisk_log_start":"69'36","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":36,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:57.185462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036671099999999998,"stat_sum":{"num_bytes":9480376,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":111,"num_read_kb":125063,"num_write":32,"num_write_kb":8571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.b","version":"72'87","reported_seq":238,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-20T02:51:29.174546+0000","last_change":"2024-04-20T02:51:24.598313+0000","last_active":"2024-04-20T02:51:29.174546+0000","last_peered":"2024-04-20T02:51:29.174546+0000","last_clean":"2024-04-20T02:51:29.174546+0000","last_became_active":"2024-04-20T02:50:35.242331+0000","last_became_peered":"2024-04-20T02:50:35.242331+0000","last_unstale":"2024-04-20T02:51:29.174546+0000","last_undegraded":"2024-04-20T02:51:29.174546+0000","last_fullsized":"2024-04-20T02:51:29.174546+0000","mapping_epoch":22,"log_start":"64'84","ondisk_log_start":"64'84","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":84,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.773051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011227373000000001,"stat_sum":{"num_bytes":16327627,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":125,"num_read_kb":145638,"num_write":69,"num_write_kb":18887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.c","version":"64'53","reported_seq":175,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-20T02:51:28.125892+0000","last_change":"2024-04-20T02:51:24.575453+0000","last_active":"2024-04-20T02:51:28.125892+0000","last_peered":"2024-04-20T02:51:28.125892+0000","last_clean":"2024-04-20T02:51:28.125892+0000","last_became_active":"2024-04-20T02:50:35.241480+0000","last_became_peered":"2024-04-20T02:50:35.241480+0000","last_unstale":"2024-04-20T02:51:28.125892+0000","last_undegraded":"2024-04-20T02:51:28.125892+0000","last_fullsized":"2024-04-20T02:51:28.125892+0000","mapping_epoch":22,"log_start":"64'49","ondisk_log_start":"64'49","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":49,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.915698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020280979999999999,"stat_sum":{"num_bytes":10579498,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":50,"num_read_kb":49915,"num_write":40,"num_write_kb":11356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.d","version":"71'98","reported_seq":270,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-20T02:51:28.494588+0000","last_change":"2024-04-20T02:51:24.575943+0000","last_active":"2024-04-20T02:51:28.494588+0000","last_peered":"2024-04-20T02:51:28.494588+0000","last_clean":"2024-04-20T02:51:28.494588+0000","last_became_active":"2024-04-20T02:50:35.528107+0000","last_became_peered":"2024-04-20T02:50:35.528107+0000","last_unstale":"2024-04-20T02:51:28.494588+0000","last_undegraded":"2024-04-20T02:51:28.494588+0000","last_fullsized":"2024-04-20T02:51:28.494588+0000","mapping_epoch":22,"log_start":"71'95","ondisk_log_start":"71'95","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":95,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:59.747989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002229221,"stat_sum":{"num_bytes":20497267,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":205,"num_read_kb":228213,"num_write":66,"num_write_kb":18181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.e","version":"71'45","reported_seq":163,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-04-20T02:51:28.164417+0000","last_change":"2024-04-20T02:51:24.575694+0000","last_active":"2024-04-20T02:51:28.164417+0000","last_peered":"2024-04-20T02:51:28.164417+0000","last_clean":"2024-04-20T02:51:28.164417+0000","last_became_active":"2024-04-20T02:50:35.244872+0000","last_became_peered":"2024-04-20T02:50:35.244872+0000","last_unstale":"2024-04-20T02:51:28.164417+0000","last_undegraded":"2024-04-20T02:51:28.164417+0000","last_fullsized":"2024-04-20T02:51:28.164417+0000","mapping_epoch":22,"log_start":"64'41","ondisk_log_start":"64'41","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":41,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:39.148211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022034310000000001,"stat_sum":{"num_bytes":9066542,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":36171,"num_write":29,"num_write_kb":10372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.f","version":"34'15","reported_seq":122,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.149182+0000","last_change":"2024-04-20T02:51:24.585406+0000","last_active":"2024-04-20T02:51:26.149182+0000","last_peered":"2024-04-20T02:51:26.149182+0000","last_clean":"2024-04-20T02:51:26.149182+0000","last_became_active":"2024-04-20T02:50:35.245323+0000","last_became_peered":"2024-04-20T02:50:35.245323+0000","last_unstale":"2024-04-20T02:51:26.149182+0000","last_undegraded":"2024-04-20T02:51:26.149182+0000","last_fullsized":"2024-04-20T02:51:26.149182+0000","mapping_epoch":22,"log_start":"33'10","ondisk_log_start":"33'10","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":10,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:53.742736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085219110000000001,"stat_sum":{"num_bytes":3428595,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":8320,"num_write":14,"num_write_kb":4030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.10","version":"65'74","reported_seq":259,"reported_epoch":69,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:26.117417+0000","last_change":"2024-04-20T02:51:21.542805+0000","last_active":"2024-04-20T02:51:26.117417+0000","last_peered":"2024-04-20T02:51:26.117417+0000","last_clean":"2024-04-20T02:51:18.527159+0000","last_became_active":"2024-04-20T02:51:21.508521+0000","last_became_peered":"2024-04-20T02:51:21.508521+0000","last_unstale":"2024-04-20T02:51:26.117417+0000","last_undegraded":"2024-04-20T02:51:26.117417+0000","last_fullsized":"2024-04-20T02:51:26.117417+0000","mapping_epoch":63,"log_start":"64'71","ondisk_log_start":"64'71","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":69,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.017870196000000001,"stat_sum":{"num_bytes":7129079,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":97,"num_read_kb":88821,"num_write":30,"num_write_kb":8248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,0,7],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.11","version":"67'112","reported_seq":342,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.580473+0000","last_change":"2024-04-20T02:51:26.580473+0000","last_active":"2024-04-20T02:51:26.580473+0000","last_peered":"2024-04-20T02:51:26.580473+0000","last_clean":"2024-04-20T02:51:26.580473+0000","last_became_active":"2024-04-20T02:51:23.559228+0000","last_became_peered":"2024-04-20T02:51:23.559228+0000","last_unstale":"2024-04-20T02:51:26.580473+0000","last_undegraded":"2024-04-20T02:51:26.580473+0000","last_fullsized":"2024-04-20T02:51:26.580473+0000","mapping_epoch":65,"log_start":"67'107","ondisk_log_start":"67'107","created":60,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":107,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.46274028299999997,"stat_sum":{"num_bytes":10584512,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":61,"num_read_kb":66428,"num_write":47,"num_write_kb":12319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":15332310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.12","version":"0'0","reported_seq":136,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-04-20T02:51:26.149193+0000","last_change":"2024-04-20T02:51:24.583777+0000","last_active":"2024-04-20T02:51:26.149193+0000","last_peered":"2024-04-20T02:51:26.149193+0000","last_clean":"2024-04-20T02:51:26.149193+0000","last_became_active":"2024-04-20T02:51:20.504935+0000","last_became_peered":"2024-04-20T02:51:20.504935+0000","last_unstale":"2024-04-20T02:51:26.149193+0000","last_undegraded":"2024-04-20T02:51:26.149193+0000","last_fullsized":"2024-04-20T02:51:26.149193+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073777629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.13","version":"72'162","reported_seq":416,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-04-20T02:51:29.164195+0000","last_change":"2024-04-20T02:51:29.137901+0000","last_active":"2024-04-20T02:51:29.164195+0000","last_peered":"2024-04-20T02:51:29.164195+0000","last_clean":"2024-04-20T02:51:29.164195+0000","last_became_active":"2024-04-20T02:51:29.136510+0000","last_became_peered":"2024-04-20T02:51:29.136510+0000","last_unstale":"2024-04-20T02:51:29.164195+0000","last_undegraded":"2024-04-20T02:51:29.164195+0000","last_fullsized":"2024-04-20T02:51:29.164195+0000","mapping_epoch":71,"log_start":"65'159","ondisk_log_start":"65'159","created":60,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":159,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088884800000000003,"stat_sum":{"num_bytes":16772086,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":115,"num_read_kb":134183,"num_write":59,"num_write_kb":15381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9140440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.14","version":"64'22","reported_seq":137,"reported_epoch":69,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:26.157848+0000","last_change":"2024-04-20T02:51:21.541565+0000","last_active":"2024-04-20T02:51:26.157848+0000","last_peered":"2024-04-20T02:51:26.157848+0000","last_clean":"2024-04-20T02:51:14.032218+0000","last_became_active":"2024-04-20T02:51:21.510278+0000","last_became_peered":"2024-04-20T02:51:21.510278+0000","last_unstale":"2024-04-20T02:51:26.157848+0000","last_undegraded":"2024-04-20T02:51:26.157848+0000","last_fullsized":"2024-04-20T02:51:26.157848+0000","mapping_epoch":63,"log_start":"56'17","ondisk_log_start":"56'17","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":17,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3093185,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":39913,"num_write":10,"num_write_kb":2705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,2],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.15","version":"64'38","reported_seq":151,"reported_epoch":65,"state":"peering","last_fresh":"2024-04-20T02:51:22.555018+0000","last_change":"2024-04-20T02:51:22.555018+0000","last_active":"2024-04-20T02:51:22.555018+0000","last_peered":"2024-04-20T02:51:21.909546+0000","last_clean":"2024-04-20T02:51:21.909546+0000","last_became_active":"2024-04-20T02:51:21.508321+0000","last_became_peered":"2024-04-20T02:51:21.508321+0000","last_unstale":"2024-04-20T02:51:22.555018+0000","last_undegraded":"2024-04-20T02:51:22.555018+0000","last_fullsized":"2024-04-20T02:51:22.555018+0000","mapping_epoch":65,"log_start":"56'33","ondisk_log_start":"56'33","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":33,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5926755,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":44474,"num_write":17,"num_write_kb":5019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6513594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.16","version":"64'61","reported_seq":265,"reported_epoch":72,"state":"remapped+peering","last_fresh":"2024-04-20T02:51:29.132040+0000","last_change":"2024-04-20T02:51:29.132206+0000","last_active":"2024-04-20T02:51:28.126565+0000","last_peered":"2024-04-20T02:51:27.811685+0000","last_clean":"2024-04-20T02:51:27.811685+0000","last_became_active":"2024-04-20T02:51:23.561843+0000","last_became_peered":"2024-04-20T02:51:23.561843+0000","last_unstale":"2024-04-20T02:51:29.132040+0000","last_undegraded":"2024-04-20T02:51:29.132040+0000","last_fullsized":"2024-04-20T02:51:29.132040+0000","mapping_epoch":72,"log_start":"59'57","ondisk_log_start":"59'57","created":60,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":55,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099853799999999999,"stat_sum":{"num_bytes":7614836,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":71,"num_read_kb":77667,"num_write":33,"num_write_kb":8547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8853601,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"1"},{"start":"19","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":214705756,"num_objects":109,"num_object_clones":63,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":6,"num_read":1499,"num_read_kb":1658605,"num_write":719,"num_write_kb":197230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":56449103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":64,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":444694528,"data_stored":545523822,"data_compressed":2032193,"data_compressed_allocated":1912832,"data_compressed_original":4815618,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":118937733,"num_objects":140,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":389,"num_read_kb":22165,"num_write":529,"num_write_kb":187742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194129920,"data_stored":210249194,"data_compressed":13031555,"data_compressed_allocated":13942784,"data_compressed_original":30674944,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509460,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103956,"kb_used_data":72832,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94267884,"statfs":{"total":96636764160,"available":96530313216,"internally_reserved":0,"allocated":74579968,"data_stored":89682062,"data_compressed":259057,"data_compressed_allocated":294912,"data_compressed_original":1011712,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.375}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.312},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.41199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.42599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.29099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.433},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.441}]}]},{"osd":5,"up_from":38,"seq":163208757256,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83336,"kb_used_data":55604,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94288504,"statfs":{"total":96636764160,"available":96551428096,"internally_reserved":0,"allocated":56938496,"data_stored":67510491,"data_compressed":1187971,"data_compressed_allocated":1273856,"data_compressed_original":2736128,"omap_allocated":1657,"internal_metadata":28375431},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68804,"kb_used_data":41784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94303036,"statfs":{"total":96636764160,"available":96566308864,"internally_reserved":0,"allocated":42786816,"data_stored":51985183,"data_compressed":995139,"data_compressed_allocated":1056768,"data_compressed_original":2269184,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.22700000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":11.877000000000001,"5min":11.877000000000001,"15min":11.877000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":11.805,"5min":11.805,"15min":11.805},"last":0.36299999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.46000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106484,"kb_used_data":75368,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94265356,"statfs":{"total":96636764160,"available":96527724544,"internally_reserved":0,"allocated":77176832,"data_stored":92874821,"data_compressed":1358695,"data_compressed_allocated":1417216,"data_compressed_original":3235414,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.436},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.46300000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.56899999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54300000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":130172,"kb_used_data":103144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94241668,"statfs":{"total":96636764160,"available":96503468032,"internally_reserved":0,"allocated":105619456,"data_stored":121294558,"data_compressed":2619929,"data_compressed_allocated":2768896,"data_compressed_original":5951062,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.32000000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28599999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.35599999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.46000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.379}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.434},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.52300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113260,"kb_used_data":82132,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94258580,"statfs":{"total":96636764160,"available":96520785920,"internally_reserved":0,"allocated":84103168,"data_stored":94723499,"data_compressed":3427450,"data_compressed_allocated":3678208,"data_compressed_original":8507392,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.58099999999999996}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.0369999999999999,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.372},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":9.9570000000000007,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.59099999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.8679999999999999,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.625},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.98,"5min":3.98,"15min":3.98},"last":0.68000000000000005}]}]},{"osd":0,"up_from":15,"seq":64424509456,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135992,"kb_used_data":104868,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94235848,"statfs":{"total":96636764160,"available":96497508352,"internally_reserved":0,"allocated":107384832,"data_stored":128964668,"data_compressed":2565341,"data_compressed_allocated":2682880,"data_compressed_original":6758400,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.51100000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.41399999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.42899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.45200000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.47599999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509458,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125616,"kb_used_data":93752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94246224,"statfs":{"total":96636764160,"available":96508133376,"internally_reserved":0,"allocated":96002048,"data_stored":112158809,"data_compressed":2655184,"data_compressed_allocated":2764800,"data_compressed_original":6426198,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.5}]},{"osd":1,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.439}]},{"osd":3,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.62}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36466688,"data_stored":39440897,"data_compressed":2216696,"data_compressed_allocated":2383872,"data_compressed_original":5459968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13307904,"data_stored":14412936,"data_compressed":977598,"data_compressed_allocated":1036288,"data_compressed_original":2203648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13307904,"data_stored":14412936,"data_compressed":977598,"data_compressed_allocated":1036288,"data_compressed_original":2203648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33099776,"data_stored":35745768,"data_compressed":2252512,"data_compressed_allocated":2404352,"data_compressed_original":5140480,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36466688,"data_stored":39440897,"data_compressed":2216696,"data_compressed_allocated":2383872,"data_compressed_original":5459968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15187968,"data_stored":16518853,"data_compressed":1153203,"data_compressed_allocated":1232896,"data_compressed_original":2605056,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46292992,"data_stored":50276907,"data_compressed":3237252,"data_compressed_allocated":3465216,"data_compressed_original":7602176,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":70053888,"data_stored":88649893,"data_compressed":343327,"data_compressed_allocated":266240,"data_compressed_original":786432,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63082496,"data_stored":78128224,"data_compressed":381097,"data_compressed_allocated":380928,"data_compressed_original":1031766,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28692480,"data_stored":37238586,"data_compressed":17541,"data_compressed_allocated":20480,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":71733248,"data_stored":85215129,"data_compressed":367417,"data_compressed_allocated":364544,"data_compressed_original":810582,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":59482112,"data_stored":73241826,"data_compressed":449424,"data_compressed_allocated":397312,"data_compressed_original":1097302,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40964096,"data_stored":50657977,"data_compressed":34768,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36970496,"data_stored":43584003,"data_compressed":184880,"data_compressed_allocated":180224,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":73715712,"data_stored":88808184,"data_compressed":253739,"data_compressed_allocated":262144,"data_compressed_original":499712,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:51:32.598 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:32.599+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:32.669 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:32.670 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:32.670 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:32.670 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:32.680 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:32.725 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:32.725+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:32.800 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:32.801+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:32.807 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:32.807 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:32.807 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:32.807 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:32.807 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:32.817 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:32.902 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:32.902+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:32.988 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:927 4.3 4:cf11ccc0:::smithi10548823-4:head [write 3121996~104915] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.610007158999998, 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058986447999999997, 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:32.989 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 927 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377548+0000", 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377607+0000", 2024-04-20T02:51:32.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9413999999999997e-05 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377616+0000", 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4880000000000007e-06 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377621+0000", 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.682e-06 2024-04-20T02:51:32.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407778+0000", 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030156110999999999 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407866+0000", 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8645000000000003e-05 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407985+0000", 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118874 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411741+0000", 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037563060000000001 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411854+0000", 2024-04-20T02:51:32.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112898 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436438+0000", 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024583885 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436483+0000", 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5334999999999999e-05 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436535+0000", 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1780000000000002e-05 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:928 4.3 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (55) in=7b,truncate 3226911] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.609928553000003, 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059064512999999999, 2024-04-20T02:51:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 928 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:32.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377630+0000", 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7259999999999998e-06 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377633+0000", 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1489999999999998e-06 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377636+0000", 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7760000000000002e-06 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:32.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410754+0000", 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033117944000000003 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410850+0000", 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6522999999999997e-05 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:33.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411008+0000", 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158157 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436151+0000", 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025142813 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436160+0000", 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1460000000000002e-06 2024-04-20T02:51:33.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436588+0000", 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00042776499999999998 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436631+0000", 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3084000000000003e-05 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436692+0000", 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0829000000000003e-05 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:929 4.3 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.609917412000001, 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059214058, 2024-04-20T02:51:33.003 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 929 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:33.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377638+0000", 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377640+0000", 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5430000000000001e-06 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377643+0000", 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0900000000000001e-06 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377647+0000", 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2230000000000001e-06 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411271+0000", 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033623635999999998 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411329+0000", 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8043e-05 2024-04-20T02:51:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436728+0000", 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025399300999999999 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436793+0000", 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5041000000000003e-05 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436853+0000", 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9446e-05 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:936 4.3 4:ca036f62:::smithi10548823-34:head [write 443718~615603] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.592882078999999, 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060880228000000002, 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.009 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 936 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394672+0000", 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394825+0000", 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015343299999999999 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394844+0000", 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8414999999999999e-05 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394859+0000", 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4738e-05 2024-04-20T02:51:33.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411879+0000", 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017020469999999999 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411954+0000", 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5239000000000002e-05 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:33.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412104+0000", 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014944399999999999 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436402+0000", 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024297828 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436409+0000", 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1779999999999997e-06 2024-04-20T02:51:33.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455390+0000", 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018981221 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455508+0000", 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117556 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455554+0000", 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6570000000000003e-05 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:937 4.3 4:ca036f62:::smithi10548823-34:head [write 1564138~470341] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.587543748999998, 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060119655000000001, 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.016 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 937 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400010+0000", 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400222+0000", 2024-04-20T02:51:33.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000211208 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400238+0000", 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419e-05 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400250+0000", 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2091e-05 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425851+0000", 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025600795999999999 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425965+0000", 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011411799999999999 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426134+0000", 2024-04-20T02:51:33.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168517 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436937+0000", 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01080328 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455351+0000", 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018414607 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460001+0000", 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046491809999999996 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460086+0000", 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5066000000000002e-05 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460132+0000", 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6174999999999998e-05 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:938 4.3 4:ca036f62:::smithi10548823-34:head [write 2633066~610652] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.580778293000002, 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054360654000000001, 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.023 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 938 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406775+0000", 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406947+0000", 2024-04-20T02:51:33.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171619 2024-04-20T02:51:33.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406962+0000", 2024-04-20T02:51:33.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5248999999999999e-05 2024-04-20T02:51:33.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406979+0000", 2024-04-20T02:51:33.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7334999999999999e-05 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436954+0000", 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029974218 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437007+0000", 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3553e-05 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437110+0000", 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455605+0000", 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018494295000000001 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455829+0000", 2024-04-20T02:51:33.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022443100000000001 2024-04-20T02:51:33.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461044+0000", 2024-04-20T02:51:33.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052150499999999997 2024-04-20T02:51:33.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461090+0000", 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6100999999999998e-05 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461138+0000", 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8003000000000001e-05 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.030 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:939 4.3 4:ca036f62:::smithi10548823-34:head [setxattr _header (56) in=7b,truncate 3243718] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.580565982000003, 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054380381999999998, 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 939 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.031 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406989+0000", 2024-04-20T02:51:33.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406993+0000", 2024-04-20T02:51:33.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9400000000000004e-06 2024-04-20T02:51:33.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406999+0000", 2024-04-20T02:51:33.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4389999999999999e-06 2024-04-20T02:51:33.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407008+0000", 2024-04-20T02:51:33.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3190000000000001e-06 2024-04-20T02:51:33.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.454982+0000", 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047973729999999999 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455051+0000", 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9197999999999997e-05 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455157+0000", 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106302 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456120+0000", 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096236599999999998 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456272+0000", 2024-04-20T02:51:33.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000152573 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461274+0000", 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005002055 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461318+0000", 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3282999999999999e-05 2024-04-20T02:51:33.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461370+0000", 2024-04-20T02:51:33.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2704999999999998e-05 2024-04-20T02:51:33.038 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.038 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.038 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.038 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:940 4.3 4:ca036f62:::smithi10548823-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.580541433, 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054543401999999998, 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 940 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.039 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407014+0000", 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407017+0000", 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9790000000000002e-06 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407022+0000", 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5539999999999997e-06 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407027+0000", 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6099999999999999e-06 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455643+0000", 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.048616331999999998 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455714+0000", 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0610000000000003e-05 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461403+0000", 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056888219999999996 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461480+0000", 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7342000000000006e-05 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461558+0000", 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7780000000000004e-05 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:51:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.442895898000003, 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.054973437000001, 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:51:33.049 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:51:33.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:51:33.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:51:33.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:51:33.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:51:33.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:51:33.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:33.056 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.054713231000001, 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:51:33.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:33.063 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.054691763000001, 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:51:33.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:51:33.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:51:33.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:51:33.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.069 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.861019831, 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.070 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:33.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:51:33.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:51:33.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.076 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:133 3.1 3:8efdf53d:::0188443c5b8894a8247535d52a547bece3a4c5f490fb220852b80e77d71ddcda:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.860295309000001, 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060974049000000002, 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127260+0000", 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127435+0000", 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017459800000000001 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127443+0000", 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3790000000000006e-06 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127450+0000", 2024-04-20T02:51:33.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5089999999999999e-06 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.129973+0000", 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025229639999999999 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130090+0000", 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117521 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130323+0000", 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023271199999999999 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.147581+0000", 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017258440999999999 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155339+0000", 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077577189999999997 2024-04-20T02:51:33.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188139+0000", 2024-04-20T02:51:33.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032800425000000001 2024-04-20T02:51:33.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188195+0000", 2024-04-20T02:51:33.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5773999999999998e-05 2024-04-20T02:51:33.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188235+0000", 2024-04-20T02:51:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9427e-05 2024-04-20T02:51:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.083 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:136 3.1 3:a965252c:::a490375a0954219328244dfa32cf9967dfac8ed04667cb9b5ce15f4910db2ee6:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.860100092000003, 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060878320999999999, 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127455+0000", 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128197+0000", 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074176999999999999 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128208+0000", 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0498e-05 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128216+0000", 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2630000000000003e-06 2024-04-20T02:51:33.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135797+0000", 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0075808669999999998 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135930+0000", 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013299000000000001 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:33.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.136192+0000", 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026213899999999999 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148053+0000", 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011860723 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155346+0000", 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0072930039999999996 2024-04-20T02:51:33.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188270+0000", 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032924299999999997 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188298+0000", 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8041000000000001e-05 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188334+0000", 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6247e-05 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:139 3.1 3:90772484:::f9b8a83c02fab1c87f6aef9574ab68b1d6a75b657e100bac3693f85cac340cb2:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.853442166999997, 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054361675999999998, 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134113+0000", 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134645+0000", 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053281300000000004 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134662+0000", 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6503000000000001e-05 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134672+0000", 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8090000000000004e-06 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148073+0000", 2024-04-20T02:51:33.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013401091 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148177+0000", 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104344 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148417+0000", 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023968899999999999 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155385+0000", 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0069679260000000002 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155725+0000", 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034046800000000002 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188383+0000", 2024-04-20T02:51:33.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032658154000000002 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188433+0000", 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0059999999999998e-05 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188475+0000", 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2049999999999999e-05 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.096 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:140 3.1 3:b80bf598:::82930be3d6fd0d013eae1f4ab54a437986489fd06a4f4934c2432c14687252ab:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.852876911999999, 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053882434, 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134678+0000", 2024-04-20T02:51:33.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134756+0000", 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7527999999999994e-05 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134763+0000", 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.092e-06 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0290000000000002e-06 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152310+0000", 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017540581999999999 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152431+0000", 2024-04-20T02:51:33.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012166 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152621+0000", 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000189944 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155992+0000", 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033703470000000001 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160642+0000", 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046505239999999996 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188508+0000", 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027866134000000001 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188534+0000", 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5945999999999999e-05 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188561+0000", 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7300999999999998e-05 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1799 4.6 4:697da67d:::smithi10548823-48:head [write 588278~715406,stat] snapc 13=[13,d] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:33.103 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.913951757, 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053216977999999998, 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1799 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073604+0000", 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073603+0000", 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:33.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073742+0000", 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000138727 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073754+0000", 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1970000000000001e-05 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073761+0000", 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.883e-06 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073806+0000", 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5840999999999997e-05 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.073872+0000", 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5520999999999998e-05 2024-04-20T02:51:33.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074005+0000", 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013345499999999999 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124313+0000", 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050307628 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124361+0000", 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7608999999999998e-05 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126705+0000", 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0023443790000000002 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126770+0000", 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4737e-05 2024-04-20T02:51:33.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.126821+0000", 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1375000000000002e-05 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1801 4.6 4:697da67d:::smithi10548823-48:head [setxattr _header (57) in=7b,truncate 2588278] snapc 13=[13,d] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.907776804000001, 2024-04-20T02:51:33.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053362723000000001, 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1801 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079782+0000", 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7840000000000001e-06 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079790+0000", 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3270000000000003e-06 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079799+0000", 2024-04-20T02:51:33.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5040000000000008e-06 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123583+0000", 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043783913000000001 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123714+0000", 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013130699999999999 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123889+0000", 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017436200000000001 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132488+0000", 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085989889999999996 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132702+0000", 2024-04-20T02:51:33.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021448600000000001 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133026+0000", 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032330500000000002 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133079+0000", 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3127000000000001e-05 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133142+0000", 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3150000000000004e-05 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1802 4.6 4:697da67d:::smithi10548823-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.907743118999999, 2024-04-20T02:51:33.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053694407999999999, 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1802 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:33.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079812+0000", 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079815+0000", 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1329999999999999e-06 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079823+0000", 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.401e-06 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079828+0000", 2024-04-20T02:51:33.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4149999999999998e-06 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124387+0000", 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044558788000000002 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124468+0000", 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1447999999999997e-05 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133186+0000", 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0087175610000000004 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133344+0000", 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158393 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133507+0000", 2024-04-20T02:51:33.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000162829 2024-04-20T02:51:33.124 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.124 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.124 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.124 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.124 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.124 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:33.126 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:33.126 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:33.126+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:33.151 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:51:33.153 INFO:tasks.rados.rados.0.smithi105.stdout:1126: done (12 left) 2024-04-20T02:51:33.153 INFO:tasks.rados.rados.0.smithi105.stdout:1127: read oid 1 snap -1 2024-04-20T02:51:33.153 INFO:tasks.rados.rados.0.smithi105.stdout:1127: expect (ObjNum 133 snap 9 seq_num 133) 2024-04-20T02:51:33.153 INFO:tasks.rados.rados.0.smithi105.stdout:1128: set_chunk oid 7 target oid 2024-04-20T02:51:33.180 INFO:tasks.rados.rados.0.smithi105.stdout:1128: set_chunk oid 7 offset: 1929728 length: 952832 target oid 48755e3149167267ef717af020e6ea4b16d2250f4f2b81c5d5d5f71807d356fe offset: 0 snap -1 2024-04-20T02:51:33.180 INFO:tasks.rados.rados.0.smithi105.stdout:1129: set_chunk oid 21 target oid 2024-04-20T02:51:33.212 INFO:tasks.rados.rados.0.smithi105.stdout:1129: set_chunk oid 21 offset: 703488 length: 1424896 target oid b4e09510887c19fc6ce537798676c1036d46bfae54a6beed99cdd6e16e434e66 offset: 0 snap -1 2024-04-20T02:51:33.212 INFO:tasks.rados.rados.0.smithi105.stdout:1130: read oid 19 snap -1 2024-04-20T02:51:33.212 INFO:tasks.rados.rados.0.smithi105.stdout:1130: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:33.212 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:33.212 INFO:tasks.rados.rados.0.smithi105.stdout:1111: finishing write tid 4 to smithi10548823-4 2024-04-20T02:51:33.212 INFO:tasks.rados.rados.0.smithi105.stdout:1111: finishing write tid 5 to smithi10548823-4 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1111: oid 4 updating version 0 to 166 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1111: oid 4 version 166 is already newer than 165 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1111: oid 4 updating version 166 to 167 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1111: oid 4 version 167 is already newer than 164 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 167 (ObjNum 191 snap 19 seq_num 191) dirty exists 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1111: left oid 4 (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1112: finishing write tid 3 to smithi10548823-46 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1112: finishing write tid 4 to smithi10548823-46 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1112: finishing write tid 5 to smithi10548823-46 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1112: oid 46 updating version 0 to 45 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1112: oid 46 updating version 45 to 46 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1112: oid 46 updating version 46 to 48 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1112: oid 46 version 48 is already newer than 47 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 48 (ObjNum 192 snap 19 seq_num 192) dirty exists 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1112: left oid 46 (ObjNum 192 snap 19 seq_num 192) 2024-04-20T02:51:33.213 INFO:tasks.rados.rados.0.smithi105.stdout:1119: got (0) Success 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 53 (ObjNum 173 snap 14 seq_num 173) dirty exists 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1117: got (0) Success 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 88 (ObjNum 155 snap 10 seq_num 155) dirty exists 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1115: finishing write tid 1 to smithi10548823-20 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1115: finishing write tid 2 to smithi10548823-20 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1115: finishing write tid 3 to smithi10548823-20 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1115: finishing write tid 4 to smithi10548823-20 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1115: finishing write tid 5 to smithi10548823-20 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1115: oid 20 updating version 0 to 92 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1115: oid 20 version 92 is already newer than 89 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1115: oid 20 version 92 is already newer than 91 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1115: oid 20 version 92 is already newer than 90 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 92 (ObjNum 193 snap 19 seq_num 193) dirty exists 2024-04-20T02:51:33.214 INFO:tasks.rados.rados.0.smithi105.stdout:1115: left oid 20 (ObjNum 193 snap 19 seq_num 193) 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1118: finishing write tid 1 to smithi10548823-6 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1118: finishing write tid 2 to smithi10548823-6 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1118: finishing write tid 3 to smithi10548823-6 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1118: finishing write tid 4 to smithi10548823-6 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1118: finishing write tid 5 to smithi10548823-6 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1118: oid 6 updating version 0 to 94 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1118: oid 6 updating version 94 to 96 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1118: oid 6 version 96 is already newer than 93 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1118: oid 6 version 96 is already newer than 95 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 96 (ObjNum 194 snap 19 seq_num 194) dirty exists 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1118: left oid 6 (ObjNum 194 snap 19 seq_num 194) 2024-04-20T02:51:33.215 INFO:tasks.rados.rados.0.smithi105.stdout:1121: expect (ObjNum 151 snap 10 seq_num 151) 2024-04-20T02:51:33.226 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:33.227+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:33.321 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:33.322 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:33.322 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:33.322 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:33.327 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:33.327+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:33.332 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:33.391 INFO:tasks.rados.rados.0.smithi105.stdout:1122: finishing write tid 1 to smithi10548823-48 2024-04-20T02:51:33.391 INFO:tasks.rados.rados.0.smithi105.stdout:1120: finishing set_chunk to oid 22 2024-04-20T02:51:33.391 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 61 (ObjNum 163 snap 14 seq_num 163) dirty exists 2024-04-20T02:51:33.391 INFO:tasks.rados.rados.0.smithi105.stdout:1122: finishing write tid 2 to smithi10548823-48 2024-04-20T02:51:33.392 INFO:tasks.rados.rados.0.smithi105.stdout:1122: finishing write tid 3 to smithi10548823-48 2024-04-20T02:51:33.392 INFO:tasks.rados.rados.0.smithi105.stdout:1122: finishing write tid 4 to smithi10548823-48 2024-04-20T02:51:33.392 INFO:tasks.rados.rados.0.smithi105.stdout:1122: finishing write tid 5 to smithi10548823-48 2024-04-20T02:51:33.392 INFO:tasks.rados.rados.0.smithi105.stdout:1122: oid 48 updating version 0 to 66 2024-04-20T02:51:33.392 INFO:tasks.rados.rados.0.smithi105.stdout:1122: oid 48 version 66 is already newer than 65 2024-04-20T02:51:33.392 INFO:tasks.rados.rados.0.smithi105.stdout:1122: oid 48 version 66 is already newer than 63 2024-04-20T02:51:33.392 INFO:tasks.rados.rados.0.smithi105.stdout:1122: oid 48 updating version 66 to 67 2024-04-20T02:51:33.392 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 67 (ObjNum 195 snap 19 seq_num 195) dirty exists 2024-04-20T02:51:33.392 INFO:tasks.rados.rados.0.smithi105.stdout:1122: left oid 48 (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:51:33.392 INFO:tasks.rados.rados.0.smithi105.stdout:1123: expect (ObjNum 181 snap 15 seq_num 181) 2024-04-20T02:51:33.427 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:33.428+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:33.503 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:33.504 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:33.504 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:33.504 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:33.504 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:33.514 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:33.528 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:33.529+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:33.596 INFO:tasks.rados.rados.0.smithi105.stdout:1124: finishing write tid 1 to smithi10548823-33 2024-04-20T02:51:33.596 INFO:tasks.rados.rados.0.smithi105.stdout:1124: finishing write tid 2 to smithi10548823-33 2024-04-20T02:51:33.596 INFO:tasks.rados.rados.0.smithi105.stdout:1111: done (15 left) 2024-04-20T02:51:33.596 INFO:tasks.rados.rados.0.smithi105.stdout:1112: done (14 left) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1115: done (13 left) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1117: done (12 left) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1118: done (11 left) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1119: done (10 left) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1120: done (9 left) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1121: done (8 left) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1122: done (7 left) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1123: done (6 left) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1131: read oid 17 snap -1 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1131: expect (ObjNum 123 snap 7 seq_num 123) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1132: read oid 49 snap -1 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1132: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1133: rollback oid 27 current snap is 19 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 27 to 13 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1124: finishing write tid 3 to smithi10548823-33 2024-04-20T02:51:33.597 INFO:tasks.rados.rados.0.smithi105.stdout:1124: finishing write tid 4 to smithi10548823-33 2024-04-20T02:51:33.598 INFO:tasks.rados.rados.0.smithi105.stdout:1124: finishing write tid 5 to smithi10548823-33 2024-04-20T02:51:33.598 INFO:tasks.rados.rados.0.smithi105.stdout:1124: oid 33 updating version 0 to 49 2024-04-20T02:51:33.598 INFO:tasks.rados.rados.0.smithi105.stdout:1124: oid 33 version 49 is already newer than 48 2024-04-20T02:51:33.598 INFO:tasks.rados.rados.0.smithi105.stdout:1124: oid 33 updating version 49 to 50 2024-04-20T02:51:33.598 INFO:tasks.rados.rados.0.smithi105.stdout:1124: oid 33 version 50 is already newer than 47 2024-04-20T02:51:33.598 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 50 (ObjNum 196 snap 19 seq_num 196) dirty exists 2024-04-20T02:51:33.598 INFO:tasks.rados.rados.0.smithi105.stdout:1124: left oid 33 (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:51:33.598 INFO:tasks.rados.rados.0.smithi105.stdout:1125: finishing set_chunk to oid 8 2024-04-20T02:51:33.598 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 29 (ObjNum 86 snap 2 seq_num 86) dirty exists 2024-04-20T02:51:33.598 INFO:tasks.rados.rados.0.smithi105.stdout:1127: expect (ObjNum 133 snap 9 seq_num 133) 2024-04-20T02:51:33.629 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:33.630+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:33.689 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.103076522999999, 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.690 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.087058245000001, 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.696 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.086905016000003, 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.702 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.707 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.086795006999999, 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.708 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:51:33.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:33.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:51:33.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.713 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.086764631000001, 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:33.718 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.082347212000002, 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:51:33.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:51:33.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:33.724 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.073095461000001, 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.070573318000001, 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.730 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:33.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:51:33.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.070488451000003, 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.736 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:51:33.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:51:33.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.742 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.070470841000002, 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:33.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:51:33.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.748 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:991 4.b e32/22 4:d4308fef:::smithi10548823-6:head v 32'29, mlcod=30'27)", 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.963566587000003, 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046400134000000003, 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724649+0000", 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724907+0000", 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025779699999999998 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724924+0000", 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7110000000000001e-05 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724934+0000", 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5370000000000003e-06 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724984+0000", 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0791000000000002e-05 2024-04-20T02:51:33.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.725011+0000", 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6299e-05 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771009+0000", 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045998144999999997 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771050+0000", 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1727999999999998e-05 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.721104232999998, 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058137664999999998, 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967112+0000", 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981895+0000", 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014783522 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981906+0000", 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0560000000000001e-05 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981912+0000", 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981956+0000", 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3455000000000003e-05 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981979+0000", 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3475000000000001e-05 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025222+0000", 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043242622000000001 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025250+0000", 2024-04-20T02:51:33.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8520000000000001e-05 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 39.299035435, 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045096171999999997, 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401391+0000", 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012209697 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401410+0000", 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8754e-05 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401422+0000", 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2089999999999999e-05 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.403142+0000", 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0017199100000000001 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.403157+0000", 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.526e-05 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434233+0000", 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031076289999999999 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434278+0000", 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4478999999999999e-05 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 39.276096244999998, 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 39.270091456999999, 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:51:33.765 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:51:33.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:130 3.0 3:1b476ba0:::474a8d6c2b8981e0c13c883e0d254815cfd155e523c9b654ff4e5007d0a1fc09:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.56075045, 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045176642000000003, 2024-04-20T02:51:33.769 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128171+0000", 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070576600000000003 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128183+0000", 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1542999999999999e-05 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128191+0000", 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5839999999999995e-06 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134592+0000", 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006400233 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134784+0000", 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000191831 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135041+0000", 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000257123 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162360+0000", 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027319774000000002 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162476+0000", 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115505 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172550+0000", 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01007366 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172598+0000", 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8216000000000001e-05 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172643+0000", 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5305000000000001e-05 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.774 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:135 3.0 3:12659911:::ad802ca609ac9a6b9a438bdacacdb4fb27f17cd228f3ff5506eb31f643d749ff:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.555722494000001, 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045928208999999998, 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132493+0000", 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133291+0000", 2024-04-20T02:51:33.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079715799999999996 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133301+0000", 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0533000000000001e-05 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133309+0000", 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9910000000000002e-06 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151891+0000", 2024-04-20T02:51:33.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018582155 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152003+0000", 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112064 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152211+0000", 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020773100000000001 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162492+0000", 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010281047999999999 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169484+0000", 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006991929 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178312+0000", 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0088282789999999996 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178374+0000", 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2123000000000001e-05 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178423+0000", 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201999999999997e-05 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.780 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:138 3.0 3:2a051c65:::f4051f504bf0c69c0b0fcc37af36dc6836e32eef2351291df998149766aaa9b9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.554901076999997, 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045257957000000001, 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:33.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133315+0000", 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133606+0000", 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029134199999999999 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133615+0000", 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2479999999999996e-06 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133621+0000", 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7739999999999999e-06 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159641+0000", 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026020008000000001 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159813+0000", 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017143200000000001 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160027+0000", 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021374400000000001 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.163217+0000", 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031905869999999999 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169517+0000", 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063002290000000001 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178484+0000", 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0089668070000000002 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178534+0000", 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9654000000000003e-05 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178574+0000", 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9929999999999999e-05 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:142 3.0 3:0f44ff3d:::ebd96dbdd0a43c00198785b2cc319b661a1b3295172ded27fbd66cbcc74da6e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.133627+0000", 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.554590079, 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045069357999999997, 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.786 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133627+0000", 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133627+0000", 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133626+0000", 2024-04-20T02:51:33.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133670+0000", 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4129999999999999e-05 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133677+0000", 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8460000000000002e-06 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133683+0000", 2024-04-20T02:51:33.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9379999999999997e-06 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162538+0000", 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028855285000000001 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162659+0000", 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000120943 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:33.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162802+0000", 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014214100000000001 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.163523+0000", 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072146799999999996 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.170143+0000", 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066200490000000002 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:33.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178613+0000", 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0084695819999999998 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178658+0000", 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.579e-05 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178696+0000", 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7768000000000002e-05 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.599583327000001, 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049605908999999997, 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088632+0000", 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090096+0000", 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001463675 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090112+0000", 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5557999999999999e-05 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090125+0000", 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3794999999999999e-05 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090196+0000", 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0537000000000006e-05 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090224+0000", 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8555999999999999e-05 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138207+0000", 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047982536999999999 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138240+0000", 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2480000000000001e-05 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.795 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.796 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.796 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:33.796 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:33.796 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:33.797 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:33.798 INFO:tasks.rados.rados.0.smithi105.stdout:1128: finishing set_chunk to oid 7 2024-04-20T02:51:33.798 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 161 (ObjNum 175 snap 14 seq_num 175) dirty exists 2024-04-20T02:51:33.798 INFO:tasks.rados.rados.0.smithi105.stdout:1129: finishing set_chunk to oid 21 2024-04-20T02:51:33.798 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk returned error code -95 offset: 703488 length: 1424896 2024-04-20T02:51:33.798 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:51:33.798 INFO:tasks.rados.rados.0.smithi105.stdout:1130: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:33.798 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:33.798+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:33.842 INFO:tasks.rados.rados.0.smithi105.stdout:1133: finishing rollback tid 0 to smithi10548823-27 2024-04-20T02:51:33.842 INFO:tasks.rados.rados.0.smithi105.stdout:1133: finishing rollback tid 1 to smithi10548823-27 2024-04-20T02:51:33.842 INFO:tasks.rados.rados.0.smithi105.stdout:1133: finishing rollback tid 2 to smithi10548823-27 2024-04-20T02:51:33.842 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 102 (ObjNum 103 snap 5 seq_num 103) dirty exists 2024-04-20T02:51:33.842 INFO:tasks.rados.rados.0.smithi105.stdout:1124: done (8 left) 2024-04-20T02:51:33.842 INFO:tasks.rados.rados.0.smithi105.stdout:1125: done (7 left) 2024-04-20T02:51:33.842 INFO:tasks.rados.rados.0.smithi105.stdout:1127: done (6 left) 2024-04-20T02:51:33.842 INFO:tasks.rados.rados.0.smithi105.stdout:1128: done (5 left) 2024-04-20T02:51:33.842 INFO:tasks.rados.rados.0.smithi105.stdout:1129: done (4 left) 2024-04-20T02:51:33.842 INFO:tasks.rados.rados.0.smithi105.stdout:1130: done (3 left) 2024-04-20T02:51:33.843 INFO:tasks.rados.rados.0.smithi105.stdout:1133: done (2 left) 2024-04-20T02:51:33.843 INFO:tasks.rados.rados.0.smithi105.stdout:1134: set_chunk oid 47 target oid 2024-04-20T02:51:33.864 INFO:tasks.rados.rados.0.smithi105.stdout:1134: set_chunk oid 47 offset: 1065472 length: 1187328 target oid 10ceaddbbb1a38b6be929742f15f4339efd359115d3459825ea55ac931a85741 offset: 0 snap -1 2024-04-20T02:51:33.864 INFO:tasks.rados.rados.0.smithi105.stdout:1135: delete oid 1 current snap is 19 2024-04-20T02:51:33.864 INFO:tasks.rados.rados.0.smithi105.stdout:1131: expect (ObjNum 123 snap 7 seq_num 123) 2024-04-20T02:51:33.897 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:33.898+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:33.968 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:33.969 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:33.969 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:33.969 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:33.979 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:33.984 INFO:tasks.rados.rados.0.smithi105.stdout:1132: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:33.999 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:33.999+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:34.099 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:34.100+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:34.114 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:34.114 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:34.114 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:34.114 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:34.114 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:34.125 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:34.131 INFO:tasks.rados.rados.0.smithi105.stdout:1134: finishing set_chunk to oid 47 2024-04-20T02:51:34.132 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 1065472 length: 1187328 2024-04-20T02:51:34.132 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:51:34.132 INFO:tasks.rados.rados.0.smithi105.stdout:1131: done (3 left) 2024-04-20T02:51:34.132 INFO:tasks.rados.rados.0.smithi105.stdout:1132: done (2 left) 2024-04-20T02:51:34.132 INFO:tasks.rados.rados.0.smithi105.stdout:1134: done (1 left) 2024-04-20T02:51:34.132 INFO:tasks.rados.rados.0.smithi105.stdout:1135: done (0 left) 2024-04-20T02:51:34.132 INFO:tasks.rados.rados.0.smithi105.stdout:1136: write oid 2 current snap is 19 2024-04-20T02:51:34.132 INFO:tasks.rados.rados.0.smithi105.stdout:1136: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-04-20T02:51:34.138 INFO:tasks.rados.rados.0.smithi105.stdout:1136: writing smithi10548823-2 from 527786 to 1252266 tid 1 2024-04-20T02:51:34.144 INFO:tasks.rados.rados.0.smithi105.stdout:1136: writing smithi10548823-2 from 1703317 to 2219371 tid 2 2024-04-20T02:51:34.150 INFO:tasks.rados.rados.0.smithi105.stdout:1136: writing smithi10548823-2 from 2804576 to 3239611 tid 3 2024-04-20T02:51:34.150 INFO:tasks.rados.rados.0.smithi105.stdout:1137: read oid 39 snap 13 2024-04-20T02:51:34.150 INFO:tasks.rados.rados.0.smithi105.stdout:1137: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:34.150 INFO:tasks.rados.rados.0.smithi105.stdout:1138: tier_promote oid 37 2024-04-20T02:51:34.150 INFO:tasks.rados.rados.0.smithi105.stdout:1139: read oid 14 snap 9 2024-04-20T02:51:34.150 INFO:tasks.rados.rados.0.smithi105.stdout:1139: expect (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:51:34.150 INFO:tasks.rados.rados.0.smithi105.stdout:1140: tier_flush oid 1 2024-04-20T02:51:34.150 INFO:tasks.rados.rados.0.smithi105.stdout:1140: tier_flush oid 1 snap -1 2024-04-20T02:51:34.150 INFO:tasks.rados.rados.0.smithi105.stdout:1141: write oid 7 current snap is 19 2024-04-20T02:51:34.150 INFO:tasks.rados.rados.0.smithi105.stdout:1141: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-04-20T02:51:34.158 INFO:tasks.rados.rados.0.smithi105.stdout:1141: writing smithi10548823-7 from 544593 to 1344322 tid 1 2024-04-20T02:51:34.165 INFO:tasks.rados.rados.0.smithi105.stdout:1141: writing smithi10548823-7 from 1761799 to 2537864 tid 2 2024-04-20T02:51:34.171 INFO:tasks.rados.rados.0.smithi105.stdout:1141: writing smithi10548823-7 from 2948352 to 3344593 tid 3 2024-04-20T02:51:34.172 INFO:tasks.rados.rados.0.smithi105.stdout:1140: got (16) Device or resource busy 2024-04-20T02:51:34.172 INFO:tasks.rados.rados.0.smithi105.stdout:check_oldest_snap_flushed oid 1 v 0 (ObjNum 3 snap 0 seq_num 304) unflushed 19 2024-04-20T02:51:34.172 INFO:tasks.rados.rados.0.smithi105.stdout:1138: got (0) Success 2024-04-20T02:51:34.172 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 38 (ObjNum 184 snap 17 seq_num 184) dirty exists 2024-04-20T02:51:34.172 INFO:tasks.rados.rados.0.smithi105.stdout:1139: expect (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:51:34.196 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:51:34.201 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:34.201+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:34.223 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:34.285 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:34.285 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.339799651, 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041399200999999997, 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945227+0000", 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947429+0000", 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022028989999999999 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947448+0000", 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.821e-05 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947460+0000", 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2373e-05 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947511+0000", 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0844999999999999e-05 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947531+0000", 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0017000000000002e-05 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986580+0000", 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039049213999999999 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986627+0000", 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7070000000000002e-05 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1145 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'30, mlcod=34'26)", 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.321068951999997, 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026752594000000001, 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.290 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963958+0000", 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:34.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964662+0000", 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070467299999999995 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964678+0000", 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5597999999999999e-05 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964687+0000", 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2979999999999997e-06 2024-04-20T02:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968184+0000", 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034972250000000001 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968211+0000", 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6829999999999999e-05 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990640+0000", 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022429317000000001 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990711+0000", 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0794e-05 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:129 3.0 e42/21 3:0256f537:::3f82a762bf2204de901e10565f70e759ee2a26bf55ffef4c8d5027a47e96ec31:head v 42'78, mlcod=42'76)", 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.150380376999998, 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024418002000000001, 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.294 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134646+0000", 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:34.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012265099999999999 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134789+0000", 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9924000000000001e-05 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134799+0000", 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0042e-05 2024-04-20T02:51:34.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134842+0000", 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3791000000000001e-05 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134866+0000", 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3791e-05 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159017+0000", 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024150919 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159065+0000", 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201000000000001e-05 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:130 3.0 e42/21 3:1b476ba0:::474a8d6c2b8981e0c13c883e0d254815cfd155e523c9b654ff4e5007d0a1fc09:head v 42'79, mlcod=42'76)", 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.149850764999996, 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023928118000000002, 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.298 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135176+0000", 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135295+0000", 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118847 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135304+0000", 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4280000000000002e-06 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135311+0000", 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4810000000000003e-06 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141973+0000", 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066622219999999998 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.142009+0000", 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5725000000000001e-05 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159073+0000", 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017064019999999999 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159105+0000", 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2081e-05 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:134 3.0 e42/21 3:33aaac3f:::d0aca4dab23a9f22fb58211e350c7c1c1b7601bafbb496b93afbbc91e2ed921c:head v 42'81, mlcod=42'76)", 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:34.302 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.140358178, 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024591152000000002, 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144949+0000", 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028058500000000002 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144959+0000", 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0335e-05 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144967+0000", 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6040000000000004e-06 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151637+0000", 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066694670000000001 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151659+0000", 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2855000000000001e-05 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169224+0000", 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017565041 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169261+0000", 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6179e-05 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1462 3.0 e44/21 3:29b349f1:::ea1112aea5e7be379214f0429a6e898cf4f4a05bdbb29b6cd15764b4776418b1:head v 44'85, mlcod=42'83)", 2024-04-20T02:51:34.306 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.107238293000002, 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034060275000000001, 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178932+0000", 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011426069999999999 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178944+0000", 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.287e-05 2024-04-20T02:51:34.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178962+0000", 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7669999999999999e-05 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179013+0000", 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1295000000000003e-05 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179038+0000", 2024-04-20T02:51:34.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4859999999999999e-05 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211813+0000", 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032775190000000003 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211850+0000", 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6216e-05 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1629 3.0 e51/21 3:088cc2f3:::a1744cc895df5e9d3e624b281fcb9edc57f3f342d1f108d4c38dc778f219a41a:head v 51'93, mlcod=45'91)", 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.481938956, 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035323188999999998, 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803088+0000", 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804187+0000", 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00109891 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804201+0000", 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3772e-05 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804213+0000", 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2335e-05 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804285+0000", 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1271000000000003e-05 2024-04-20T02:51:34.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804306+0000", 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1359999999999999e-05 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838377+0000", 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034070994 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838412+0000", 2024-04-20T02:51:34.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4790999999999999e-05 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1653 4.6 e51/22 4:697da67d:::smithi10548823-48:head v 51'41, mlcod=45'37)", 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.860199848000001, 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037653640000000002, 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424827+0000", 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425066+0000", 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023957399999999999 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425117+0000", 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1082999999999999e-05 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425146+0000", 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8129000000000001e-05 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425217+0000", 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1402000000000005e-05 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425278+0000", 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1498e-05 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462456+0000", 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037177106000000001 2024-04-20T02:51:34.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462481+0000", 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5851e-05 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1786 4.d e56/22 4:b376c20e:::smithi10548823-49:head v 56'80, mlcod=55'77)", 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.473108311000001, 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028136330000000001, 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.319 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:34.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812644+0000", 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072510100000000002 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812656+0000", 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1688999999999999e-05 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812663+0000", 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:34.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812707+0000", 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4573999999999997e-05 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812722+0000", 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5319e-05 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840021+0000", 2024-04-20T02:51:34.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027298111999999999 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840056+0000", 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5086000000000003e-05 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1799 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'53, mlcod=54'49)", 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.210779290000001, 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050527214000000001, 2024-04-20T02:51:34.323 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074614+0000", 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036631499999999998 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074624+0000", 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8330000000000006e-06 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074632+0000", 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3599999999999998e-06 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074683+0000", 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1137999999999997e-05 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074704+0000", 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1616000000000001e-05 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124714+0000", 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050009642 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124776+0000", 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1581000000000006e-05 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:34.327 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.196283193999999, 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046184680999999998, 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088743+0000", 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090327+0000", 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001584071 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090343+0000", 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5193e-05 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090354+0000", 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1336e-05 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090411+0000", 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6589999999999999e-05 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090431+0000", 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0579000000000001e-05 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134883+0000", 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044451877000000001 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134929+0000", 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6131000000000003e-05 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1800 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'54, mlcod=54'49)", 2024-04-20T02:51:34.331 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.183877119999998, 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027029681999999999, 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101150+0000", 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101284+0000", 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000134688 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101295+0000", 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101305+0000", 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0930999999999999e-05 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101347+0000", 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1522e-05 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101369+0000", 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2027000000000001e-05 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128138+0000", 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026768626 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128180+0000", 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2568999999999998e-05 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1807 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'87, mlcod=56'82)", 2024-04-20T02:51:34.335 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 20.176196294, 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026347280000000001, 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108830+0000", 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109221+0000", 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039087699999999998 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109239+0000", 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8904000000000001e-05 2024-04-20T02:51:34.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109271+0000", 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1319000000000003e-05 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120423+0000", 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011151912999999999 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120454+0000", 2024-04-20T02:51:34.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1464999999999998e-05 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135158+0000", 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014703554000000001 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135179+0000", 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0965999999999999e-05 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2038 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'159, mlcod=0'0)", 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.7034077209999996, 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044306238999999997, 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581964+0000", 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034420099999999998 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581980+0000", 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6578999999999998e-05 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581991+0000", 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0949e-05 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582734+0000", 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074277299999999996 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582761+0000", 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6627000000000001e-05 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625882+0000", 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043121617000000001 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625926+0000", 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3838999999999998e-05 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.343 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2039 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'160, mlcod=0'0)", 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.6830096359999995, 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027318719000000002, 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602017+0000", 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602364+0000", 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000347139 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602375+0000", 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.128e-05 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602384+0000", 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2600000000000005e-06 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605223+0000", 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0028393400000000001 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605245+0000", 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2078000000000001e-05 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629297+0000", 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024051605 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629337+0000", 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0126000000000002e-05 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.347 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2137 3.0 e70/21 3:0d26903d:::7146376437549e0b99264d159bf1e9c8bc43ac1c7eb519f65899cfd592a43a4a:head v 70'98, mlcod=55'96)", 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.5008816520000003, 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024394852000000002, 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784144+0000", 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785280+0000", 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001135741 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785299+0000", 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8845999999999999e-05 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785310+0000", 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1450999999999999e-05 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785362+0000", 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1798000000000001e-05 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785388+0000", 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5763999999999999e-05 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808507+0000", 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023118742000000001 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808541+0000", 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4258000000000003e-05 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.351 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2142 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'65, mlcod=66'61)", 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.4681822389999999, 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033365672999999998, 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816981+0000", 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135952 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816994+0000", 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3405e-05 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817003+0000", 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4319999999999999e-06 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817055+0000", 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1842999999999998e-05 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817076+0000", 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1375000000000001e-05 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850179+0000", 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033103051000000001 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850211+0000", 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1896000000000001e-05 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.355 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2143 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'66, mlcod=66'61)", 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.4543163970000004, 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023949208999999999, 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830710+0000", 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830883+0000", 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017331000000000001 2024-04-20T02:51:34.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830903+0000", 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0043000000000001e-05 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830914+0000", 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1263999999999999e-05 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.836558+0000", 2024-04-20T02:51:34.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056437309999999999 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.836587+0000", 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.854633+0000", 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01804584 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.854660+0000", 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7708e-05 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.143647624, 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039421115999999999, 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141379+0000", 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142797+0000", 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001418702 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142816+0000", 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8907999999999999e-05 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142825+0000", 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1020000000000003e-06 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142872+0000", 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6168999999999998e-05 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142891+0000", 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9561000000000001e-05 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180704+0000", 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037812643 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180801+0000", 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7361999999999994e-05 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2280 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'105, mlcod=73'100)", 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.122966347, 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029657706999999998, 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:34.364 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162060+0000", 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:34.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162455+0000", 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039569799999999999 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162472+0000", 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6820000000000002e-05 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162485+0000", 2024-04-20T02:51:34.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2388999999999999e-05 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163758+0000", 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001273906 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163784+0000", 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5352000000000001e-05 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191663+0000", 2024-04-20T02:51:34.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027879583999999999 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191719+0000", 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640000000000003e-05 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:34.368 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:34.370 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:34.370 INFO:tasks.rados.rados.0.smithi105.stdout:1137: expect (ObjNum 127 snap 9 seq_num 127) 2024-04-20T02:51:34.370 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:34.370+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:34.398 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:34.419 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:34.420 INFO:tasks.rados.rados.0.smithi105.stdout:1136: finishing write tid 1 to smithi10548823-2 2024-04-20T02:51:34.420 INFO:tasks.rados.rados.0.smithi105.stdout:1136: finishing write tid 2 to smithi10548823-2 2024-04-20T02:51:34.420 INFO:tasks.rados.rados.0.smithi105.stdout:1136: finishing write tid 3 to smithi10548823-2 2024-04-20T02:51:34.420 INFO:tasks.rados.rados.0.smithi105.stdout:1136: finishing write tid 4 to smithi10548823-2 2024-04-20T02:51:34.420 INFO:tasks.rados.rados.0.smithi105.stdout:1136: finishing write tid 5 to smithi10548823-2 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1136: oid 2 updating version 0 to 105 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1136: oid 2 version 105 is already newer than 104 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1136: oid 2 updating version 105 to 107 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1136: oid 2 version 107 is already newer than 106 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 107 (ObjNum 198 snap 19 seq_num 198) dirty exists 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1136: left oid 2 (ObjNum 198 snap 19 seq_num 198) 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1141: finishing write tid 1 to smithi10548823-7 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1141: finishing write tid 2 to smithi10548823-7 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1141: finishing write tid 3 to smithi10548823-7 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1141: finishing write tid 4 to smithi10548823-7 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1141: finishing write tid 5 to smithi10548823-7 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1141: oid 7 updating version 0 to 165 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1141: oid 7 updating version 165 to 166 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1141: oid 7 updating version 166 to 167 2024-04-20T02:51:34.421 INFO:tasks.rados.rados.0.smithi105.stdout:1141: oid 7 version 167 is already newer than 164 2024-04-20T02:51:34.422 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 167 (ObjNum 199 snap 19 seq_num 199) dirty exists 2024-04-20T02:51:34.422 INFO:tasks.rados.rados.0.smithi105.stdout:1141: left oid 7 (ObjNum 199 snap 19 seq_num 199) 2024-04-20T02:51:34.422 INFO:tasks.rados.rados.0.smithi105.stdout:1136: done (5 left) 2024-04-20T02:51:34.422 INFO:tasks.rados.rados.0.smithi105.stdout:1137: done (4 left) 2024-04-20T02:51:34.422 INFO:tasks.rados.rados.0.smithi105.stdout:1138: done (3 left) 2024-04-20T02:51:34.422 INFO:tasks.rados.rados.0.smithi105.stdout:1139: done (2 left) 2024-04-20T02:51:34.422 INFO:tasks.rados.rados.0.smithi105.stdout:1140: done (1 left) 2024-04-20T02:51:34.422 INFO:tasks.rados.rados.0.smithi105.stdout:1141: done (0 left) 2024-04-20T02:51:34.422 INFO:tasks.rados.rados.0.smithi105.stdout:1142: snap_remove snap 6 2024-04-20T02:51:34.440 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:34.459 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:34.471 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:34.471+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:34.479 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:34.500 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:34.521 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:34.532 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:34.532 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:34.541 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:34.543 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:34.543 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:34.543 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:34.543 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:34.543 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:34.543 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:34.543 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:34.544 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:34.544 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:34.596 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:34.595+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:34.672 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:34.672+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:34.722 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:34.722 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:34.732 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:34.774 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:34.774+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:34.874 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:34.875+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:34.901 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:34.901 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:34.912 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:34.974 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:34.975+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:35.053 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:35.054 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:35.064 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:35.105 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:35.106+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:35.176 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:35.177+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:35.180 INFO:tasks.rados.rados.0.smithi105.stdout:1142: done (0 left) 2024-04-20T02:51:35.180 INFO:tasks.rados.rados.0.smithi105.stdout:1143: read oid 47 snap -1 2024-04-20T02:51:35.180 INFO:tasks.rados.rados.0.smithi105.stdout:1143: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:51:35.180 INFO:tasks.rados.rados.0.smithi105.stdout:1144: read oid 1 snap 11 2024-04-20T02:51:35.180 INFO:tasks.rados.rados.0.smithi105.stdout:1144: expect (ObjNum 133 snap 9 seq_num 133) 2024-04-20T02:51:35.180 INFO:tasks.rados.rados.0.smithi105.stdout:1145: read oid 4 snap -1 2024-04-20T02:51:35.180 INFO:tasks.rados.rados.0.smithi105.stdout:1145: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:51:35.180 INFO:tasks.rados.rados.0.smithi105.stdout:1146: write (excl) oid 28 current snap is 19 2024-04-20T02:51:35.180 INFO:tasks.rados.rados.0.smithi105.stdout:1146: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-04-20T02:51:35.186 INFO:tasks.rados.rados.0.smithi105.stdout:1146: writing smithi10548823-28 from 561400 to 1036378 tid 1 2024-04-20T02:51:35.194 INFO:tasks.rados.rados.0.smithi105.stdout:1146: writing smithi10548823-28 from 1820281 to 2340004 tid 2 2024-04-20T02:51:35.200 INFO:tasks.rados.rados.0.smithi105.stdout:1146: writing smithi10548823-28 from 2859422 to 3317001 tid 3 2024-04-20T02:51:35.200 INFO:tasks.rados.rados.0.smithi105.stdout:1147: tier_promote oid 34 2024-04-20T02:51:35.200 INFO:tasks.rados.rados.0.smithi105.stdout:1144: expect (ObjNum 133 snap 9 seq_num 133) 2024-04-20T02:51:35.267 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:35.267 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:35.277 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:35.277+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:35.278 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:35.333 INFO:tasks.rados.rados.0.smithi105.stdout:1145: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:51:35.378 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:35.378+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:35.406 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:35.406 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:35.416 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:35.479 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:35.479+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:35.577 INFO:tasks.rados.rados.0.smithi105.stdout:1143: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:51:35.580 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:35.581+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:35.585 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:35.585 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:35.595 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:1147: got (0) Success 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 157 (ObjNum 170 snap 14 seq_num 170) dirty exists 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:1146: finishing write tid 1 to smithi10548823-28 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:1146: finishing write tid 2 to smithi10548823-28 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:1143: done (4 left) 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:1144: done (3 left) 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:1145: done (2 left) 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:1147: done (1 left) 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:1148: read oid 3 snap -1 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:1148: expect deleted 2024-04-20T02:51:35.668 INFO:tasks.rados.rados.0.smithi105.stdout:1149: snap_remove snap 13 2024-04-20T02:51:35.680 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:35.681+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:35.765 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:35.765 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:35.775 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:35.781 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:35.782+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:35.882 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:35.883+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:35.903 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:35.903 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:35.914 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:35.997 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:35.998+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:36.083 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:36.083 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:36.084 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:36.084+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:36.093 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:36.122 INFO:tasks.rados.rados.0.smithi105.stdout:1149: done (2 left) 2024-04-20T02:51:36.122 INFO:tasks.rados.rados.0.smithi105.stdout:1150: tier_promote oid 5 2024-04-20T02:51:36.122 INFO:tasks.rados.rados.0.smithi105.stdout:1151: read oid 42 snap -1 2024-04-20T02:51:36.122 INFO:tasks.rados.rados.0.smithi105.stdout:1151: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:51:36.122 INFO:tasks.rados.rados.0.smithi105.stdout:1152: read oid 14 snap -1 2024-04-20T02:51:36.123 INFO:tasks.rados.rados.0.smithi105.stdout:1152: expect (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:51:36.123 INFO:tasks.rados.rados.0.smithi105.stdout:1153: write (excl) oid 29 current snap is 19 2024-04-20T02:51:36.123 INFO:tasks.rados.rados.0.smithi105.stdout:1153: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-04-20T02:51:36.127 INFO:tasks.rados.rados.0.smithi105.stdout:1153: writing smithi10548823-29 from 578207 to 1128434 tid 1 2024-04-20T02:51:36.136 INFO:tasks.rados.rados.0.smithi105.stdout:1153: writing smithi10548823-29 from 1762410 to 2542144 tid 2 2024-04-20T02:51:36.141 INFO:tasks.rados.rados.0.smithi105.stdout:1153: writing smithi10548823-29 from 3286845 to 3378207 tid 3 2024-04-20T02:51:36.141 INFO:tasks.rados.rados.0.smithi105.stdout:1154: delete oid 41 current snap is 19 2024-04-20T02:51:36.141 INFO:tasks.rados.rados.0.smithi105.stdout:1146: finishing write tid 3 to smithi10548823-28 2024-04-20T02:51:36.141 INFO:tasks.rados.rados.0.smithi105.stdout:1146: finishing write tid 4 to smithi10548823-28 2024-04-20T02:51:36.141 INFO:tasks.rados.rados.0.smithi105.stdout:1146: finishing write tid 5 to smithi10548823-28 2024-04-20T02:51:36.142 INFO:tasks.rados.rados.0.smithi105.stdout:1146: oid 28 updating version 0 to 126 2024-04-20T02:51:36.142 INFO:tasks.rados.rados.0.smithi105.stdout:1146: oid 28 version 126 is already newer than 124 2024-04-20T02:51:36.142 INFO:tasks.rados.rados.0.smithi105.stdout:1146: oid 28 version 126 is already newer than 125 2024-04-20T02:51:36.142 INFO:tasks.rados.rados.0.smithi105.stdout:1146: oid 28 version 126 is already newer than 123 2024-04-20T02:51:36.142 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 126 (ObjNum 200 snap 19 seq_num 200) dirty exists 2024-04-20T02:51:36.142 INFO:tasks.rados.rados.0.smithi105.stdout:1146: left oid 28 (ObjNum 200 snap 19 seq_num 200) 2024-04-20T02:51:36.142 INFO:tasks.rados.rados.0.smithi105.stdout:1152: expect (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:51:36.184 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:36.184+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:36.210 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:51:36.266 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:36.266 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:36.276 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:36.315 INFO:tasks.rados.rados.0.smithi105.stdout:1150: got (0) Success 2024-04-20T02:51:36.315 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 116 (ObjNum 186 snap 18 seq_num 186) dirty exists 2024-04-20T02:51:36.315 INFO:tasks.rados.rados.0.smithi105.stdout:1151: expect (ObjNum 108 snap 6 seq_num 108) 2024-04-20T02:51:36.317 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:36.317+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:36.387 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:36.387+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:36.445 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:36.445 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:36.457 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:36.491 INFO:tasks.rados.rados.0.smithi105.stdout:1153: finishing write tid 1 to smithi10548823-29 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1153: finishing write tid 2 to smithi10548823-29 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1153: finishing write tid 3 to smithi10548823-29 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1153: finishing write tid 4 to smithi10548823-29 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1153: finishing write tid 5 to smithi10548823-29 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1153: oid 29 updating version 0 to 19 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1153: oid 29 version 19 is already newer than 18 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1153: oid 29 updating version 19 to 20 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1153: oid 29 version 20 is already newer than 17 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 20 (ObjNum 201 snap 19 seq_num 201) dirty exists 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1153: left oid 29 (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1146: done (6 left) 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1148: done (5 left) 2024-04-20T02:51:36.492 INFO:tasks.rados.rados.0.smithi105.stdout:1150: done (4 left) 2024-04-20T02:51:36.493 INFO:tasks.rados.rados.0.smithi105.stdout:1151: done (3 left) 2024-04-20T02:51:36.493 INFO:tasks.rados.rados.0.smithi105.stdout:1152: done (2 left) 2024-04-20T02:51:36.493 INFO:tasks.rados.rados.0.smithi105.stdout:1153: done (1 left) 2024-04-20T02:51:36.493 INFO:tasks.rados.rados.0.smithi105.stdout:1154: done (0 left) 2024-04-20T02:51:36.493 INFO:tasks.rados.rados.0.smithi105.stdout:1155: tier_flush oid 40 2024-04-20T02:51:36.493 INFO:tasks.rados.rados.0.smithi105.stdout:1155: tier_flush oid 40 snap -1 2024-04-20T02:51:36.493 INFO:tasks.rados.rados.0.smithi105.stdout:1156: snap_remove snap 11 2024-04-20T02:51:36.500 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:36.500+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:36.555 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:36.586 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:36.586 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:36.597 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:36.598 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:36.599+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:36.690 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:36.689+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:36.698 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:36.717 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:36.735 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:36.754 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:36.765 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:36.765 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:36.773 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:36.776 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:36.791 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:36.791+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:36.791 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:36.810 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:36.830 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:36.831 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:36.892 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:36.893+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:36.945 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.945 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.0.15:152 3.2 e48/47 3:54f0c00e:::2528189ec1c891f179c539fa23af2c87b9e8d2c19f70e70fac8934d3cc51ccee:head v 48'96, mlcod=0'0)", 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.511846229, 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020196797999999998, 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:36.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.432906+0000", 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433091+0000", 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000185589 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433102+0000", 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.433108+0000", 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5660000000000003e-06 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439137+0000", 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060284329999999997 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.439166+0000", 2024-04-20T02:51:36.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9278e-05 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453075+0000", 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013909163 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:05.453103+0000", 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7725999999999998e-05 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1635 3.2 e51/49 3:67667c16:::7aea211498f946c134ed12b1f5632511a2f083a0eb2114d1051f38dc7326b113:head v 51'100, mlcod=0'0)", 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.097262595, 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019392829, 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847490+0000", 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.847489+0000", 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850819+0000", 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033292069999999998 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850831+0000", 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2320000000000001e-05 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850838+0000", 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2980000000000003e-06 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850874+0000", 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5800999999999998e-05 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:36.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.850890+0000", 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5536000000000002e-05 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866849+0000", 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015959346999999999 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:08.866883+0000", 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3776999999999999e-05 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.496114356, 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.957 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1665 4.1 e52/46 4:8b91c759:::smithi10548823-5:head v 52'78, mlcod=50'73)", 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.421514503000001, 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025918594, 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523238+0000", 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523236+0000", 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523658+0000", 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000422082 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523676+0000", 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7716000000000001e-05 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523686+0000", 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9790000000000005e-06 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523734+0000", 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7579e-05 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.523759+0000", 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5593e-05 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549122+0000", 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025363087999999999 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.549157+0000", 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4242999999999997e-05 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.961 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 26.489606124000002, 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034212370999999998, 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:36.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456260+0000", 2024-04-20T02:51:36.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011140480000000001 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456273+0000", 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456280+0000", 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1130000000000003e-06 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456331+0000", 2024-04-20T02:51:36.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1446999999999997e-05 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456357+0000", 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6001999999999999e-05 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489309+0000", 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032951972000000003 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489359+0000", 2024-04-20T02:51:36.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9552000000000003e-05 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1775 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'41, mlcod=53'38)", 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.191224634000001, 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038672274999999999, 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754868+0000", 2024-04-20T02:51:36.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013404619999999999 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754881+0000", 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.323e-05 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754892+0000", 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1154e-05 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754967+0000", 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4326999999999995e-05 2024-04-20T02:51:36.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.755006+0000", 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9338000000000002e-05 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792154+0000", 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037147924999999998 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792200+0000", 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6044999999999997e-05 2024-04-20T02:51:36.969 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1776 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'42, mlcod=53'38)", 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.171244456, 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019186657999999999, 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773508+0000", 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.773507+0000", 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774271+0000", 2024-04-20T02:51:36.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076432799999999999 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774284+0000", 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2483000000000001e-05 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774294+0000", 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6569999999999992e-06 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774338+0000", 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4576000000000001e-05 2024-04-20T02:51:36.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.774358+0000", 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9449999999999998e-05 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792664+0000", 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018306413000000001 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792695+0000", 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0685000000000002e-05 2024-04-20T02:51:36.973 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1822 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'94, mlcod=55'92)", 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.841202501000001, 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031823907999999998, 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.974 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103786+0000", 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000236658 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103797+0000", 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0951000000000001e-05 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103804+0000", 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0960000000000003e-06 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103860+0000", 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5781e-05 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103887+0000", 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6999999999999999e-05 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135333+0000", 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031446004 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135374+0000", 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0778999999999997e-05 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1823 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'95, mlcod=55'92)", 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.826715494999998, 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017980995999999999, 2024-04-20T02:51:36.978 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118037+0000", 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118036+0000", 2024-04-20T02:51:36.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118363+0000", 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00032734600000000001 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118381+0000", 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7816e-05 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.118391+0000", 2024-04-20T02:51:36.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0155e-05 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119591+0000", 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011995560000000001 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.119618+0000", 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7475e-05 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135988+0000", 2024-04-20T02:51:36.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016370096000000001 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.136018+0000", 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9745999999999999e-05 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1836 4.1 e58/46 4:8b927dd6:::smithi10548823-28:head v 58'99, mlcod=56'96)", 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 21.459186493000001, 2024-04-20T02:51:36.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030374581000000001, 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485930+0000", 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00036455800000000002 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485941+0000", 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1017e-05 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485950+0000", 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.85e-06 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486000+0000", 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9589999999999998e-05 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486025+0000", 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.529e-05 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515909+0000", 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029884131000000001 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515941+0000", 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1569000000000002e-05 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:36.986 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.445931589000001, 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.990 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.44586614, 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.445840642, 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:36.995 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1891 4.17 e61/60 4:e9876a63:::smithi10548823-30:head v 61'45, mlcod=0'0)", 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:36.999 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.431294846, 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018777245000000001, 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513458+0000", 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.513457+0000", 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514596+0000", 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011393289999999999 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514612+0000", 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6311e-05 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514623+0000", 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056999999999999e-05 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514670+0000", 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6557999999999997e-05 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.514698+0000", 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8121000000000001e-05 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.532204+0000", 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017505969999999999 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.532235+0000", 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0731000000000001e-05 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1892 4.17 e61/60 4:e9876a63:::smithi10548823-30:head v 61'46, mlcod=0'0)", 2024-04-20T02:51:37.003 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.407955862000001, 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020303921999999999, 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536797+0000", 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.536796+0000", 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537578+0000", 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00078217499999999997 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537592+0000", 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3721e-05 2024-04-20T02:51:37.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537602+0000", 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0380000000000001e-05 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537638+0000", 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5559000000000003e-05 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.537657+0000", 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9242e-05 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.557064+0000", 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019406970999999999 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.557101+0000", 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6572000000000002e-05 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2004 4.13 e65/63 4:cfd1e4d4:::smithi10548823-9:head v 65'159, mlcod=64'156)", 2024-04-20T02:51:37.007 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.096615593999999, 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036995865000000003, 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849186+0000", 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010492889999999999 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849197+0000", 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1365999999999999e-05 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849204+0000", 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1069999999999999e-06 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849250+0000", 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5213999999999998e-05 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849279+0000", 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9149000000000002e-05 2024-04-20T02:51:37.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885093+0000", 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035813921999999998 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885133+0000", 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0077999999999998e-05 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2005 4.13 e65/63 4:cfd1e4d4:::smithi10548823-9:head v 65'160, mlcod=64'156)", 2024-04-20T02:51:37.011 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.076754627, 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.01736478, 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867998+0000", 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.867997+0000", 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868547+0000", 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000550287 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868563+0000", 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5676e-05 2024-04-20T02:51:37.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.868572+0000", 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0440000000000001e-06 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.874844+0000", 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0062724460000000001 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.874865+0000", 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0712000000000001e-05 2024-04-20T02:51:37.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885334+0000", 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010468691 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885363+0000", 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8923000000000001e-05 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.015 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2029 3.2 e66/49 3:70b121b2:::b13b8a8d7724a2b6951a53e14e5e568e1aa45f9c18b0373ab4a92ed71f345848:head v 66'106, mlcod=53'104)", 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:24.119464+0000", 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.825288909999999, 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023218222, 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.119464+0000", 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.119464+0000", 2024-04-20T02:51:37.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.119462+0000", 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122123+0000", 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002661629 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122140+0000", 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6290000000000002e-05 2024-04-20T02:51:37.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122147+0000", 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.695e-06 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122201+0000", 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.393e-05 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.122228+0000", 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6367999999999999e-05 2024-04-20T02:51:37.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.142651+0000", 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020423884 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:24.142682+0000", 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0301999999999999e-05 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.019 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2292 4.3 e76/60 4:c07c6daa:::smithi10548823-7:head v 76'164, mlcod=64'161)", 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:34.164410+0000", 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 2.7803429350000002, 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022413272000000001, 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.164410+0000", 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.164410+0000", 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.164405+0000", 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999957 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.165495+0000", 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010896429999999999 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.165520+0000", 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5034000000000001e-05 2024-04-20T02:51:37.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.165534+0000", 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3763e-05 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.165598+0000", 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4289999999999996e-05 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.165630+0000", 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1780999999999999e-05 2024-04-20T02:51:37.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.186767+0000", 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.021137824999999999 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.186823+0000", 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5405000000000002e-05 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2293 4.3 e76/60 4:c07c6daa:::smithi10548823-7:head v 76'165, mlcod=64'161)", 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:34.188793+0000", 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 2.7559597679999999, 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018061721999999999, 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.188793+0000", 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.188793+0000", 2024-04-20T02:51:37.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.188792+0000", 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189614+0000", 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00082219499999999996 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189629+0000", 2024-04-20T02:51:37.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4987999999999999e-05 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189638+0000", 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8829999999999995e-06 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189677+0000", 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9150999999999998e-05 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189696+0000", 2024-04-20T02:51:37.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9225e-05 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.206816+0000", 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017119737999999999 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.206854+0000", 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8371999999999998e-05 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:37.027 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:37.028 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:37.029 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:37.029 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:37.029+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:37.125 INFO:tasks.rados.rados.0.smithi105.stdout:1156: done (1 left) 2024-04-20T02:51:37.126 INFO:tasks.rados.rados.0.smithi105.stdout:1157: read oid 36 snap -1 2024-04-20T02:51:37.126 INFO:tasks.rados.rados.0.smithi105.stdout:1157: expect (ObjNum 190 snap 18 seq_num 190) 2024-04-20T02:51:37.126 INFO:tasks.rados.rados.0.smithi105.stdout:1158: read oid 27 snap -1 2024-04-20T02:51:37.126 INFO:tasks.rados.rados.0.smithi105.stdout:1158: expect (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:51:37.126 INFO:tasks.rados.rados.0.smithi105.stdout:1159: read oid 29 snap 15 2024-04-20T02:51:37.126 INFO:tasks.rados.rados.0.smithi105.stdout:1159: expect (ObjNum 85 snap 2 seq_num 85) 2024-04-20T02:51:37.126 INFO:tasks.rados.rados.0.smithi105.stdout:1160: set_chunk oid 15 target oid 2024-04-20T02:51:37.130 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:37.129+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:37.135 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:51:37.168 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:37.168 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:37.168 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:37.169 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:37.179 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:37.263 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:37.263+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:37.331 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:37.331+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:37.346 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:37.346 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:37.346 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:37.346 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:37.346 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:37.356 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:37.442 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:37.441+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:37.506 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:51:37.506 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:51:37.506 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-20T02:51:37.507 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.978420 2024-04-20T02:51:37.507 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 0.9784203218927009 2024-04-20T02:51:37.529 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:927 4.3 4:cf11ccc0:::smithi10548823-4:head [write 3121996~104915] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:37.529 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.150008376999999, 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058986447999999997, 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 927 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377548+0000", 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377607+0000", 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9413999999999997e-05 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377616+0000", 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4880000000000007e-06 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377621+0000", 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.682e-06 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407778+0000", 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030156110999999999 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407866+0000", 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8645000000000003e-05 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407985+0000", 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118874 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411741+0000", 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037563060000000001 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411854+0000", 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112898 2024-04-20T02:51:37.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436438+0000", 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024583885 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436483+0000", 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5334999999999999e-05 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436535+0000", 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1780000000000002e-05 2024-04-20T02:51:37.535 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:928 4.3 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (55) in=7b,truncate 3226911] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.149929770999996, 2024-04-20T02:51:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059064512999999999, 2024-04-20T02:51:37.536 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 928 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377630+0000", 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7259999999999998e-06 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377633+0000", 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1489999999999998e-06 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377636+0000", 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7760000000000002e-06 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410754+0000", 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033117944000000003 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410850+0000", 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6522999999999997e-05 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411008+0000", 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158157 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436151+0000", 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025142813 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436160+0000", 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1460000000000002e-06 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436588+0000", 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00042776499999999998 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436631+0000", 2024-04-20T02:51:37.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3084000000000003e-05 2024-04-20T02:51:37.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.542 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436692+0000", 2024-04-20T02:51:37.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0829000000000003e-05 2024-04-20T02:51:37.542 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.542 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.542 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:929 4.3 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.149918630000002, 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059214058, 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 929 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377638+0000", 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377640+0000", 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5430000000000001e-06 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377643+0000", 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0900000000000001e-06 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377647+0000", 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2230000000000001e-06 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411271+0000", 2024-04-20T02:51:37.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033623635999999998 2024-04-20T02:51:37.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:37.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411329+0000", 2024-04-20T02:51:37.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8043e-05 2024-04-20T02:51:37.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436728+0000", 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025399300999999999 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436793+0000", 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5041000000000003e-05 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436853+0000", 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9446e-05 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.547 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:936 4.3 4:ca036f62:::smithi10548823-34:head [write 443718~615603] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.132883296999999, 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060880228000000002, 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.548 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 936 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394672+0000", 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394825+0000", 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015343299999999999 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394844+0000", 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8414999999999999e-05 2024-04-20T02:51:37.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394859+0000", 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4738e-05 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411879+0000", 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017020469999999999 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411954+0000", 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5239000000000002e-05 2024-04-20T02:51:37.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:37.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412104+0000", 2024-04-20T02:51:37.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014944399999999999 2024-04-20T02:51:37.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436402+0000", 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024297828 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436409+0000", 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1779999999999997e-06 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455390+0000", 2024-04-20T02:51:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018981221 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455508+0000", 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117556 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455554+0000", 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6570000000000003e-05 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:937 4.3 4:ca036f62:::smithi10548823-34:head [write 1564138~470341] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.127544966999999, 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060119655000000001, 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 937 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400010+0000", 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400222+0000", 2024-04-20T02:51:37.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000211208 2024-04-20T02:51:37.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400238+0000", 2024-04-20T02:51:37.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419e-05 2024-04-20T02:51:37.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400250+0000", 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2091e-05 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425851+0000", 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025600795999999999 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425965+0000", 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011411799999999999 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426134+0000", 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168517 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436937+0000", 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01080328 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455351+0000", 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018414607 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460001+0000", 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046491809999999996 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460086+0000", 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5066000000000002e-05 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460132+0000", 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6174999999999998e-05 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.561 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.562 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.562 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:938 4.3 4:ca036f62:::smithi10548823-34:head [write 2633066~610652] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:37.562 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:37.562 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.120779511000002, 2024-04-20T02:51:37.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054360654000000001, 2024-04-20T02:51:37.562 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.562 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.562 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 938 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406775+0000", 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406947+0000", 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171619 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406962+0000", 2024-04-20T02:51:37.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5248999999999999e-05 2024-04-20T02:51:37.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406979+0000", 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7334999999999999e-05 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436954+0000", 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029974218 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437007+0000", 2024-04-20T02:51:37.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3553e-05 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437110+0000", 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455605+0000", 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018494295000000001 2024-04-20T02:51:37.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455829+0000", 2024-04-20T02:51:37.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022443100000000001 2024-04-20T02:51:37.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461044+0000", 2024-04-20T02:51:37.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052150499999999997 2024-04-20T02:51:37.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461090+0000", 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6100999999999998e-05 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461138+0000", 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8003000000000001e-05 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:939 4.3 4:ca036f62:::smithi10548823-34:head [setxattr _header (56) in=7b,truncate 3243718] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:37.570 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:37.571 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.120567200000004, 2024-04-20T02:51:37.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054380381999999998, 2024-04-20T02:51:37.571 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.571 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.571 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.571 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.571 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.571 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 939 2024-04-20T02:51:37.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.573 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:37.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406989+0000", 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406993+0000", 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9400000000000004e-06 2024-04-20T02:51:37.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406999+0000", 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4389999999999999e-06 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407008+0000", 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3190000000000001e-06 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.454982+0000", 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047973729999999999 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455051+0000", 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9197999999999997e-05 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455157+0000", 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106302 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456120+0000", 2024-04-20T02:51:37.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096236599999999998 2024-04-20T02:51:37.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456272+0000", 2024-04-20T02:51:37.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000152573 2024-04-20T02:51:37.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461274+0000", 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005002055 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461318+0000", 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3282999999999999e-05 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461370+0000", 2024-04-20T02:51:37.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2704999999999998e-05 2024-04-20T02:51:37.579 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.579 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.579 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.579 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:940 4.3 4:ca036f62:::smithi10548823-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:37.579 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:37.579 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.120542651000001, 2024-04-20T02:51:37.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054543401999999998, 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 940 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407014+0000", 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407017+0000", 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9790000000000002e-06 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407022+0000", 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5539999999999997e-06 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407027+0000", 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6099999999999999e-06 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455643+0000", 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.048616331999999998 2024-04-20T02:51:37.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455714+0000", 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0610000000000003e-05 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461403+0000", 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056888219999999996 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461480+0000", 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7342000000000006e-05 2024-04-20T02:51:37.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461558+0000", 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7780000000000004e-05 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.982897115999997, 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.584 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:37.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:51:37.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:51:37.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.594974655000001, 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.588 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.589 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.589 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.589 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.589 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:51:37.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.589 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:37.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:51:37.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:51:37.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:51:37.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:51:37.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:51:37.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:37.595 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.594714449000001, 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:51:37.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:51:37.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:51:37.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:51:37.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:51:37.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:51:37.601 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.594692981000001, 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.602 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.401021049000001, 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:51:37.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:51:37.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:133 3.1 3:8efdf53d:::0188443c5b8894a8247535d52a547bece3a4c5f490fb220852b80e77d71ddcda:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.400296527000002, 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060974049000000002, 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.613 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127260+0000", 2024-04-20T02:51:37.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127435+0000", 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017459800000000001 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127443+0000", 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3790000000000006e-06 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127450+0000", 2024-04-20T02:51:37.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5089999999999999e-06 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.129973+0000", 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025229639999999999 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130090+0000", 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117521 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:37.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130323+0000", 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023271199999999999 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.147581+0000", 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017258440999999999 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155339+0000", 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077577189999999997 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188139+0000", 2024-04-20T02:51:37.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032800425000000001 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188195+0000", 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5773999999999998e-05 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188235+0000", 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9427e-05 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:136 3.1 3:a965252c:::a490375a0954219328244dfa32cf9967dfac8ed04667cb9b5ce15f4910db2ee6:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.400101309999997, 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060878320999999999, 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:37.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127455+0000", 2024-04-20T02:51:37.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128197+0000", 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074176999999999999 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128208+0000", 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0498e-05 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128216+0000", 2024-04-20T02:51:37.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2630000000000003e-06 2024-04-20T02:51:37.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135797+0000", 2024-04-20T02:51:37.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0075808669999999998 2024-04-20T02:51:37.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135930+0000", 2024-04-20T02:51:37.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013299000000000001 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.136192+0000", 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026213899999999999 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148053+0000", 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011860723 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155346+0000", 2024-04-20T02:51:37.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0072930039999999996 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188270+0000", 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032924299999999997 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188298+0000", 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8041000000000001e-05 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188334+0000", 2024-04-20T02:51:37.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6247e-05 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:139 3.1 3:90772484:::f9b8a83c02fab1c87f6aef9574ab68b1d6a75b657e100bac3693f85cac340cb2:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.393443384999998, 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054361675999999998, 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134113+0000", 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134645+0000", 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053281300000000004 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134662+0000", 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6503000000000001e-05 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134672+0000", 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8090000000000004e-06 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148073+0000", 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013401091 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148177+0000", 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104344 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148417+0000", 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023968899999999999 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155385+0000", 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0069679260000000002 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155725+0000", 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034046800000000002 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188383+0000", 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032658154000000002 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188433+0000", 2024-04-20T02:51:37.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0059999999999998e-05 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188475+0000", 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2049999999999999e-05 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:140 3.1 3:b80bf598:::82930be3d6fd0d013eae1f4ab54a437986489fd06a4f4934c2432c14687252ab:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:37.630 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.39287813, 2024-04-20T02:51:37.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053882434, 2024-04-20T02:51:37.631 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.631 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.631 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:37.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:37.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134678+0000", 2024-04-20T02:51:37.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134756+0000", 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7527999999999994e-05 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134763+0000", 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.092e-06 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0290000000000002e-06 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152310+0000", 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017540581999999999 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152431+0000", 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012166 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152621+0000", 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000189944 2024-04-20T02:51:37.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155992+0000", 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033703470000000001 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160642+0000", 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046505239999999996 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188508+0000", 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027866134000000001 2024-04-20T02:51:37.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188534+0000", 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5945999999999999e-05 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188561+0000", 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7300999999999998e-05 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.636 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1801 4.6 4:697da67d:::smithi10548823-48:head [setxattr _header (57) in=7b,truncate 2588278] snapc 13=[13,d] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.447778022000001, 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053362723000000001, 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1801 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:37.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079782+0000", 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7840000000000001e-06 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079790+0000", 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3270000000000003e-06 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079799+0000", 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5040000000000008e-06 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123583+0000", 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043783913000000001 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123714+0000", 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013130699999999999 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123889+0000", 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017436200000000001 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132488+0000", 2024-04-20T02:51:37.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085989889999999996 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132702+0000", 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021448600000000001 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133026+0000", 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032330500000000002 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133079+0000", 2024-04-20T02:51:37.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3127000000000001e-05 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133142+0000", 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3150000000000004e-05 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1802 4.6 4:697da67d:::smithi10548823-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.447744337, 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053694407999999999, 2024-04-20T02:51:37.642 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1802 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079812+0000", 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079815+0000", 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1329999999999999e-06 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079823+0000", 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.401e-06 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079828+0000", 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4149999999999998e-06 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124387+0000", 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044558788000000002 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:37.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124468+0000", 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1447999999999997e-05 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133186+0000", 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0087175610000000004 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133344+0000", 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158393 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133507+0000", 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000162829 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2292 4.3 4:c07c6daa:::smithi10548823-7:head [write 544593~799729] snapc 1e=[1e,1d,1b,18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.3679778570000001, 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061161747000000002, 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:37.647 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2292 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:37.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159577+0000", 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160019+0000", 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000442674 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160045+0000", 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5446e-05 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:37.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160058+0000", 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3444e-05 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160487+0000", 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000428899 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160616+0000", 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012861099999999999 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160838+0000", 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000222415 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207115+0000", 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046276543000000003 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207164+0000", 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9497999999999999e-05 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220657+0000", 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013492386 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220698+0000", 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1451e-05 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220741+0000", 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3078000000000003e-05 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.652 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.653 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:37.653 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:37.653 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:37.654 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:37.655 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:37.654+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:37.754 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:37.755+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:37.824 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:37.825 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:37.825 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:37.825 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:37.835 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:37.855 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:37.855+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:37.956 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:37.956+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:37.962 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:37.962 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:37.962 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:37.962 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:37.962 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:37.974 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:38.056 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:38.057+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:38.127 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.4 primary-affinity to 0.97842 (8641212) 2024-04-20T02:51:38.148 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:38.148 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:38.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:38.148 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:38.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.148 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:38.148 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.562589948999999, 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:51:38.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:51:38.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:51:38.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:51:38.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.546571671000002, 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:51:38.154 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.546418441999997, 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:38.160 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:38.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:51:38.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:51:38.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:51:38.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:51:38.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:51:38.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:51:38.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:51:38.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:51:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:51:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.546308433, 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:51:38.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:51:38.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:51:38.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.546278057000002, 2024-04-20T02:51:38.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:51:38.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:38.180 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.541860638000003, 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:51:38.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:51:38.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:51:38.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:51:38.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.532608887000002, 2024-04-20T02:51:38.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:51:38.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:38.192 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.530086744000002, 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:38.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:38.198 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.530001876999997, 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:38.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:51:38.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:51:38.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.204 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.529984267000003, 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:38.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:51:38.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:51:38.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:991 4.b e32/22 4:d4308fef:::smithi10548823-6:head v 32'29, mlcod=30'27)", 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.423080013000003, 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046400134000000003, 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724649+0000", 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724907+0000", 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025779699999999998 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724924+0000", 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7110000000000001e-05 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724934+0000", 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5370000000000003e-06 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724984+0000", 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0791000000000002e-05 2024-04-20T02:51:38.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.725011+0000", 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6299e-05 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771009+0000", 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045998144999999997 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771050+0000", 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1727999999999998e-05 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.213 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.180617658999999, 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058137664999999998, 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:38.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967112+0000", 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981895+0000", 2024-04-20T02:51:38.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014783522 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981906+0000", 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0560000000000001e-05 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981912+0000", 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981956+0000", 2024-04-20T02:51:38.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3455000000000003e-05 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981979+0000", 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3475000000000001e-05 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025222+0000", 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043242622000000001 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025250+0000", 2024-04-20T02:51:38.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8520000000000001e-05 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.758548861000001, 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045096171999999997, 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401391+0000", 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012209697 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401410+0000", 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8754e-05 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401422+0000", 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2089999999999999e-05 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.403142+0000", 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0017199100000000001 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.403157+0000", 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.526e-05 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434233+0000", 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031076289999999999 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434278+0000", 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4478999999999999e-05 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.735609670999999, 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.222 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:38.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:51:38.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.729604883, 2024-04-20T02:51:38.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:51:38.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.230 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:130 3.0 3:1b476ba0:::474a8d6c2b8981e0c13c883e0d254815cfd155e523c9b654ff4e5007d0a1fc09:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.020263876000001, 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045176642000000003, 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128171+0000", 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070576600000000003 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128183+0000", 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1542999999999999e-05 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128191+0000", 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5839999999999995e-06 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134592+0000", 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006400233 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134784+0000", 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000191831 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135041+0000", 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000257123 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162360+0000", 2024-04-20T02:51:38.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027319774000000002 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162476+0000", 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115505 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172550+0000", 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01007366 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172598+0000", 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8216000000000001e-05 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172643+0000", 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5305000000000001e-05 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:135 3.0 3:12659911:::ad802ca609ac9a6b9a438bdacacdb4fb27f17cd228f3ff5506eb31f643d749ff:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.015235920000002, 2024-04-20T02:51:38.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045928208999999998, 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132493+0000", 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133291+0000", 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079715799999999996 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133301+0000", 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0533000000000001e-05 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133309+0000", 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9910000000000002e-06 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151891+0000", 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018582155 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152003+0000", 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112064 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152211+0000", 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020773100000000001 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162492+0000", 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010281047999999999 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169484+0000", 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006991929 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178312+0000", 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0088282789999999996 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178374+0000", 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2123000000000001e-05 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178423+0000", 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201999999999997e-05 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.241 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:138 3.0 3:2a051c65:::f4051f504bf0c69c0b0fcc37af36dc6836e32eef2351291df998149766aaa9b9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.014414502999998, 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045257957000000001, 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:38.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133315+0000", 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133606+0000", 2024-04-20T02:51:38.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029134199999999999 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133615+0000", 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2479999999999996e-06 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133621+0000", 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7739999999999999e-06 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159641+0000", 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026020008000000001 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159813+0000", 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017143200000000001 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160027+0000", 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021374400000000001 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.163217+0000", 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031905869999999999 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169517+0000", 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063002290000000001 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178484+0000", 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0089668070000000002 2024-04-20T02:51:38.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178534+0000", 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9654000000000003e-05 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178574+0000", 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9929999999999999e-05 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.247 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.059096752999999, 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049605908999999997, 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088632+0000", 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090096+0000", 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001463675 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090112+0000", 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5557999999999999e-05 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090125+0000", 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3794999999999999e-05 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090196+0000", 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0537000000000006e-05 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090224+0000", 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8555999999999999e-05 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138207+0000", 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047982536999999999 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138240+0000", 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2480000000000001e-05 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.251 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.0063887600000001, 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056137746000000002, 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141340+0000", 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142751+0000", 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014113540000000001 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142770+0000", 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8450000000000001e-05 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142780+0000", 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0284e-05 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142982+0000", 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000202244 2024-04-20T02:51:38.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.143007+0000", 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4929000000000001e-05 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197446+0000", 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054438519999999997 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197479+0000", 2024-04-20T02:51:38.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3597999999999999e-05 2024-04-20T02:51:38.256 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.256 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.256 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.256 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.256 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.256 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:38.258 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:38.258 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:38.257+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:38.359 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:38.359+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:38.426 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:38.427 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:38.427 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:38.427 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:38.437 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:38.479 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:38.478+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:38.561 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:38.561+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:38.606 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:38.606 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:38.606 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:38.606 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:38.606 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:38.617 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:38.662 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:38.662+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.801200057999999, 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041399200999999997, 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945227+0000", 2024-04-20T02:51:38.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947429+0000", 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022028989999999999 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947448+0000", 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.821e-05 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947460+0000", 2024-04-20T02:51:38.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2373e-05 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947511+0000", 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0844999999999999e-05 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947531+0000", 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0017000000000002e-05 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986580+0000", 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039049213999999999 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986627+0000", 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7070000000000002e-05 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1145 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'30, mlcod=34'26)", 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.782469358999997, 2024-04-20T02:51:38.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026752594000000001, 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963958+0000", 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964662+0000", 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070467299999999995 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964678+0000", 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5597999999999999e-05 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964687+0000", 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2979999999999997e-06 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968184+0000", 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034972250000000001 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968211+0000", 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6829999999999999e-05 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990640+0000", 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022429317000000001 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990711+0000", 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0794e-05 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:129 3.0 e42/21 3:0256f537:::3f82a762bf2204de901e10565f70e759ee2a26bf55ffef4c8d5027a47e96ec31:head v 42'78, mlcod=42'76)", 2024-04-20T02:51:38.755 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.611780783999997, 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024418002000000001, 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134646+0000", 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012265099999999999 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134789+0000", 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9924000000000001e-05 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134799+0000", 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0042e-05 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134842+0000", 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3791000000000001e-05 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134866+0000", 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3791e-05 2024-04-20T02:51:38.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159017+0000", 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024150919 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159065+0000", 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201000000000001e-05 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.759 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:130 3.0 e42/21 3:1b476ba0:::474a8d6c2b8981e0c13c883e0d254815cfd155e523c9b654ff4e5007d0a1fc09:head v 42'79, mlcod=42'76)", 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.611251172000003, 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023928118000000002, 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:38.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135176+0000", 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135295+0000", 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118847 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135304+0000", 2024-04-20T02:51:38.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4280000000000002e-06 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135311+0000", 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4810000000000003e-06 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141973+0000", 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066622219999999998 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.142009+0000", 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5725000000000001e-05 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159073+0000", 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017064019999999999 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159105+0000", 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2081e-05 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:134 3.0 e42/21 3:33aaac3f:::d0aca4dab23a9f22fb58211e350c7c1c1b7601bafbb496b93afbbc91e2ed921c:head v 42'81, mlcod=42'76)", 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.601758584999999, 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024591152000000002, 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144949+0000", 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028058500000000002 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144959+0000", 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0335e-05 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144967+0000", 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6040000000000004e-06 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151637+0000", 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066694670000000001 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151659+0000", 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2855000000000001e-05 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169224+0000", 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017565041 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169261+0000", 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6179e-05 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1462 3.0 e44/21 3:29b349f1:::ea1112aea5e7be379214f0429a6e898cf4f4a05bdbb29b6cd15764b4776418b1:head v 44'85, mlcod=42'83)", 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.568638700000001, 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034060275000000001, 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:38.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:38.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178932+0000", 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011426069999999999 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178944+0000", 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.287e-05 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178962+0000", 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7669999999999999e-05 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179013+0000", 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1295000000000003e-05 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179038+0000", 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4859999999999999e-05 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211813+0000", 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032775190000000003 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211850+0000", 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6216e-05 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1629 3.0 e51/21 3:088cc2f3:::a1744cc895df5e9d3e624b281fcb9edc57f3f342d1f108d4c38dc778f219a41a:head v 51'93, mlcod=45'91)", 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.943339363, 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035323188999999998, 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803088+0000", 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804187+0000", 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00109891 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804201+0000", 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3772e-05 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804213+0000", 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2335e-05 2024-04-20T02:51:38.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804285+0000", 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1271000000000003e-05 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804306+0000", 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1359999999999999e-05 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838377+0000", 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034070994 2024-04-20T02:51:38.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838412+0000", 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4790999999999999e-05 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1653 4.6 e51/22 4:697da67d:::smithi10548823-48:head v 51'41, mlcod=45'37)", 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.321600255, 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037653640000000002, 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.776 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424827+0000", 2024-04-20T02:51:38.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425066+0000", 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023957399999999999 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425117+0000", 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1082999999999999e-05 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425146+0000", 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8129000000000001e-05 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425217+0000", 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1402000000000005e-05 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425278+0000", 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1498e-05 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462456+0000", 2024-04-20T02:51:38.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037177106000000001 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462481+0000", 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5851e-05 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1786 4.d e56/22 4:b376c20e:::smithi10548823-49:head v 56'80, mlcod=55'77)", 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.934508718, 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028136330000000001, 2024-04-20T02:51:38.780 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812644+0000", 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072510100000000002 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812656+0000", 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1688999999999999e-05 2024-04-20T02:51:38.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812663+0000", 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812707+0000", 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4573999999999997e-05 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812722+0000", 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5319e-05 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840021+0000", 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027298111999999999 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840056+0000", 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5086000000000003e-05 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1799 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'53, mlcod=54'49)", 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.672179697000001, 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050527214000000001, 2024-04-20T02:51:38.785 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074614+0000", 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036631499999999998 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074624+0000", 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8330000000000006e-06 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074632+0000", 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3599999999999998e-06 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074683+0000", 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1137999999999997e-05 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074704+0000", 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1616000000000001e-05 2024-04-20T02:51:38.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124714+0000", 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050009642 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124776+0000", 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1581000000000006e-05 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.789 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.657683600999999, 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046184680999999998, 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:38.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088743+0000", 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090327+0000", 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001584071 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090343+0000", 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5193e-05 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090354+0000", 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1336e-05 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090411+0000", 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6589999999999999e-05 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090431+0000", 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0579000000000001e-05 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134883+0000", 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044451877000000001 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134929+0000", 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6131000000000003e-05 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1800 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'54, mlcod=54'49)", 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.645277527000001, 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027029681999999999, 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.794 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101150+0000", 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101284+0000", 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000134688 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101295+0000", 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101305+0000", 2024-04-20T02:51:38.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0930999999999999e-05 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101347+0000", 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1522e-05 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101369+0000", 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2027000000000001e-05 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128138+0000", 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026768626 2024-04-20T02:51:38.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128180+0000", 2024-04-20T02:51:38.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2568999999999998e-05 2024-04-20T02:51:38.798 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.798 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.798 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.798 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1807 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'87, mlcod=56'82)", 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 24.637596701, 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026347280000000001, 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108830+0000", 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109221+0000", 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039087699999999998 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109239+0000", 2024-04-20T02:51:38.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8904000000000001e-05 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109271+0000", 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1319000000000003e-05 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120423+0000", 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011151912999999999 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120454+0000", 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1464999999999998e-05 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135158+0000", 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014703554000000001 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135179+0000", 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0965999999999999e-05 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.802 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2038 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'159, mlcod=0'0)", 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.164808128000001, 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044306238999999997, 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581964+0000", 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034420099999999998 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581980+0000", 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6578999999999998e-05 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581991+0000", 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0949e-05 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582734+0000", 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074277299999999996 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582761+0000", 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6627000000000001e-05 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625882+0000", 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043121617000000001 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625926+0000", 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3838999999999998e-05 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.806 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2039 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'160, mlcod=0'0)", 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.144410043000001, 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027318719000000002, 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602017+0000", 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602364+0000", 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000347139 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602375+0000", 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.128e-05 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602384+0000", 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2600000000000005e-06 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605223+0000", 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0028393400000000001 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605245+0000", 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2078000000000001e-05 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629297+0000", 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024051605 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629337+0000", 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0126000000000002e-05 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2137 3.0 e70/21 3:0d26903d:::7146376437549e0b99264d159bf1e9c8bc43ac1c7eb519f65899cfd592a43a4a:head v 70'98, mlcod=55'96)", 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.962282059, 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024394852000000002, 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784144+0000", 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785280+0000", 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001135741 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785299+0000", 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8845999999999999e-05 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785310+0000", 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1450999999999999e-05 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785362+0000", 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1798000000000001e-05 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785388+0000", 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5763999999999999e-05 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808507+0000", 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023118742000000001 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808541+0000", 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4258000000000003e-05 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2142 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'65, mlcod=66'61)", 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.929582646, 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033365672999999998, 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.815 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:38.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816981+0000", 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135952 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816994+0000", 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3405e-05 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817003+0000", 2024-04-20T02:51:38.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4319999999999999e-06 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817055+0000", 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1842999999999998e-05 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817076+0000", 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1375000000000001e-05 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850179+0000", 2024-04-20T02:51:38.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033103051000000001 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850211+0000", 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1896000000000001e-05 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2143 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'66, mlcod=66'61)", 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.915716804000001, 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023949208999999999, 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.819 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830710+0000", 2024-04-20T02:51:38.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830883+0000", 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017331000000000001 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830903+0000", 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0043000000000001e-05 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830914+0000", 2024-04-20T02:51:38.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1263999999999999e-05 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.836558+0000", 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056437309999999999 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.836587+0000", 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.854633+0000", 2024-04-20T02:51:38.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01804584 2024-04-20T02:51:38.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.854660+0000", 2024-04-20T02:51:38.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7708e-05 2024-04-20T02:51:38.823 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.823 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.823 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.6050480309999999, 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039421115999999999, 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141379+0000", 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142797+0000", 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001418702 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142816+0000", 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8907999999999999e-05 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142825+0000", 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1020000000000003e-06 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142872+0000", 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6168999999999998e-05 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142891+0000", 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9561000000000001e-05 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180704+0000", 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037812643 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180801+0000", 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7361999999999994e-05 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.827 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2280 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'105, mlcod=73'100)", 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.5843667540000004, 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029657706999999998, 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162060+0000", 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162455+0000", 2024-04-20T02:51:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039569799999999999 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162472+0000", 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6820000000000002e-05 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162485+0000", 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2388999999999999e-05 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163758+0000", 2024-04-20T02:51:38.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001273906 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163784+0000", 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5352000000000001e-05 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191663+0000", 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027879583999999999 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191719+0000", 2024-04-20T02:51:38.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640000000000003e-05 2024-04-20T02:51:38.832 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.832 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.832 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.832 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:38.832 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:38.832 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:38.833 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:38.834 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:38.834+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:38.842 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:38.934 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:38.934+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:39.008 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:39.008 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:39.008 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:39.009 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:39.019 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:39.022 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:39.035 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:39.036+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:39.042 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:39.062 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:39.083 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:39.103 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:39.123 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:39.136 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:39.135+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:39.151 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:51:39.152 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:39.158 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:39.158 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:39.170 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:39.173 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:39.175 INFO:tasks.rados.rados.0.smithi105.stdout:1160: set_chunk oid 15 offset: 903168 length: 1517056 target oid 5e8c8cfb7872ba3d906a04cbc88776822ecc3fcc5f2c8dfa9e0a7deabeee647c offset: 0 snap -1 2024-04-20T02:51:39.175 INFO:tasks.rados.rados.0.smithi105.stdout:1161: set_chunk oid 25 target oid 2024-04-20T02:51:39.175 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:39.175 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:39.175 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:39.175 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:39.176 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:39.176 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:39.176 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:39.176 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:39.214 INFO:tasks.rados.rados.0.smithi105.stdout:1161: set_chunk oid 25 offset: 1214976 length: 1285632 target oid 019f2a4a04d4abde904f07a760a84ad9ef5a6a028bf2143539619aba9fb68250 offset: 0 snap -1 2024-04-20T02:51:39.214 INFO:tasks.rados.rados.0.smithi105.stdout:1162: read oid 10 snap -1 2024-04-20T02:51:39.214 INFO:tasks.rados.rados.0.smithi105.stdout:1162: expect (ObjNum 156 snap 10 seq_num 156) 2024-04-20T02:51:39.214 INFO:tasks.rados.rados.0.smithi105.stdout:1163: write oid 2 current snap is 19 2024-04-20T02:51:39.214 INFO:tasks.rados.rados.0.smithi105.stdout:1163: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-04-20T02:51:39.221 INFO:tasks.rados.rados.0.smithi105.stdout:1163: writing smithi10548823-2 from 611821 to 1312546 tid 1 2024-04-20T02:51:39.228 INFO:tasks.rados.rados.0.smithi105.stdout:1163: writing smithi10548823-2 from 1879374 to 2662777 tid 2 2024-04-20T02:51:39.232 INFO:tasks.rados.rados.0.smithi105.stdout:1163: writing smithi10548823-2 from 3341691 to 3411821 tid 3 2024-04-20T02:51:39.232 INFO:tasks.rados.rados.0.smithi105.stdout:1164: tier_evict oid 38 2024-04-20T02:51:39.233 INFO:tasks.rados.rados.0.smithi105.stdout:1164: tier_evict oid 38 snap -1 2024-04-20T02:51:39.233 INFO:tasks.rados.rados.0.smithi105.stdout:1159: expect (ObjNum 85 snap 2 seq_num 85) 2024-04-20T02:51:39.237 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:39.237+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:39.335 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:39.335 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:39.337 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:39.337+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:39.346 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:39.437 INFO:tasks.rados.rados.0.smithi105.stdout:1158: expect (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:51:39.438 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:39.437+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:39.476 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:39.476 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:39.487 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:39.571 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:39.571+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:39.636 INFO:tasks.rados.rados.0.smithi105.stdout:1155: got (16) Device or resource busy 2024-04-20T02:51:39.636 INFO:tasks.rados.rados.0.smithi105.stdout:check_oldest_snap_flushed oid 40 v 168 (ObjNum 0 snap 0 seq_num 304) unflushed 18 2024-04-20T02:51:39.636 INFO:tasks.rados.rados.0.smithi105.stdout:1157: expect (ObjNum 190 snap 18 seq_num 190) 2024-04-20T02:51:39.639 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:39.639+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:39.656 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:39.656 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:39.667 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:39.742 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:39.742+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:39.825 INFO:tasks.rados.rados.0.smithi105.stdout:1160: finishing set_chunk to oid 15 2024-04-20T02:51:39.825 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 164 (ObjNum 157 snap 11 seq_num 157) dirty exists 2024-04-20T02:51:39.825 INFO:tasks.rados.rados.0.smithi105.stdout:1155: done (8 left) 2024-04-20T02:51:39.825 INFO:tasks.rados.rados.0.smithi105.stdout:1157: done (7 left) 2024-04-20T02:51:39.825 INFO:tasks.rados.rados.0.smithi105.stdout:1158: done (6 left) 2024-04-20T02:51:39.825 INFO:tasks.rados.rados.0.smithi105.stdout:1159: done (5 left) 2024-04-20T02:51:39.825 INFO:tasks.rados.rados.0.smithi105.stdout:1160: done (4 left) 2024-04-20T02:51:39.826 INFO:tasks.rados.rados.0.smithi105.stdout:1165: write oid 8 current snap is 19 2024-04-20T02:51:39.826 INFO:tasks.rados.rados.0.smithi105.stdout:1161: finishing set_chunk to oid 25 2024-04-20T02:51:39.826 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 85 (ObjNum 169 snap 14 seq_num 169) dirty exists 2024-04-20T02:51:39.826 INFO:tasks.rados.rados.0.smithi105.stdout:1162: expect (ObjNum 156 snap 10 seq_num 156) 2024-04-20T02:51:39.835 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:39.835 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:39.843 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:39.843+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:39.845 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:39.911 INFO:tasks.rados.rados.0.smithi105.stdout:1164: got (22) Invalid argument 2024-04-20T02:51:39.911 INFO:tasks.rados.rados.0.smithi105.stdout:1163: finishing write tid 1 to smithi10548823-2 2024-04-20T02:51:39.911 INFO:tasks.rados.rados.0.smithi105.stdout:1163: finishing write tid 2 to smithi10548823-2 2024-04-20T02:51:39.911 INFO:tasks.rados.rados.0.smithi105.stdout:1163: finishing write tid 3 to smithi10548823-2 2024-04-20T02:51:39.911 INFO:tasks.rados.rados.0.smithi105.stdout:1163: finishing write tid 4 to smithi10548823-2 2024-04-20T02:51:39.912 INFO:tasks.rados.rados.0.smithi105.stdout:1165: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-04-20T02:51:39.918 INFO:tasks.rados.rados.0.smithi105.stdout:1165: writing smithi10548823-8 from 628628 to 1404602 tid 1 2024-04-20T02:51:39.923 INFO:tasks.rados.rados.0.smithi105.stdout:1165: writing smithi10548823-8 from 1937856 to 2464917 tid 2 2024-04-20T02:51:39.929 INFO:tasks.rados.rados.0.smithi105.stdout:1165: writing smithi10548823-8 from 3252761 to 3428628 tid 3 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1161: done (4 left) 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1162: done (3 left) 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1164: done (2 left) 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1166: tier_flush oid 46 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1166: tier_flush oid 46 snap -1 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1163: finishing write tid 5 to smithi10548823-2 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1163: oid 2 updating version 0 to 126 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1163: oid 2 updating version 126 to 127 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1163: oid 2 version 127 is already newer than 124 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1163: oid 2 version 127 is already newer than 125 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 127 (ObjNum 203 snap 19 seq_num 203) dirty exists 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1163: left oid 2 (ObjNum 203 snap 19 seq_num 203) 2024-04-20T02:51:39.930 INFO:tasks.rados.rados.0.smithi105.stdout:1163: done (2 left) 2024-04-20T02:51:39.931 INFO:tasks.rados.rados.0.smithi105.stdout:1167: set_chunk oid 34 target oid 2024-04-20T02:51:39.931 INFO:tasks.rados.rados.0.smithi105.stdout:1167: set_chunk oid 34 offset: 2256384 length: 1485824 target oid smithi10548823- offset: 2256384 snap 12 2024-04-20T02:51:39.931 INFO:tasks.rados.rados.0.smithi105.stdout:1168: read oid 41 snap 15 2024-04-20T02:51:39.931 INFO:tasks.rados.rados.0.smithi105.stdout:1168: expect (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:51:39.931 INFO:tasks.rados.rados.0.smithi105.stdout:1169: write oid 24 current snap is 19 2024-04-20T02:51:39.931 INFO:tasks.rados.rados.0.smithi105.stdout:1169: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-04-20T02:51:39.935 INFO:tasks.rados.rados.0.smithi105.stdout:1169: writing smithi10548823-24 from 645435 to 1096658 tid 1 2024-04-20T02:51:39.943 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:39.943+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:39.944 INFO:tasks.rados.rados.0.smithi105.stdout:1169: writing smithi10548823-24 from 1879985 to 2667057 tid 2 2024-04-20T02:51:39.948 INFO:tasks.rados.rados.0.smithi105.stdout:1169: writing smithi10548823-24 from 3280184 to 3445435 tid 3 2024-04-20T02:51:39.948 INFO:tasks.rados.rados.0.smithi105.stdout:1170: write (excl) oid 30 current snap is 19 2024-04-20T02:51:39.949 INFO:tasks.rados.rados.0.smithi105.stdout:1167: finishing set_chunk to oid 34 2024-04-20T02:51:39.949 INFO:tasks.rados.rados.0.smithi105.stdout:1167: got expected ENOENT (src dne) 2024-04-20T02:51:39.949 INFO:tasks.rados.rados.0.smithi105.stdout:1170: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-04-20T02:51:39.955 INFO:tasks.rados.rados.0.smithi105.stdout:1170: writing smithi10548823-30 from 662242 to 1305067 tid 1 2024-04-20T02:51:39.962 INFO:tasks.rados.rados.0.smithi105.stdout:1170: writing smithi10548823-30 from 2054820 to 2585550 tid 2 2024-04-20T02:51:39.968 INFO:tasks.rados.rados.0.smithi105.stdout:1170: writing smithi10548823-30 from 3307607 to 3462242 tid 3 2024-04-20T02:51:39.968 INFO:tasks.rados.rados.0.smithi105.stdout:1167: done (5 left) 2024-04-20T02:51:39.968 INFO:tasks.rados.rados.0.smithi105.stdout:1171: read oid 38 snap -1 2024-04-20T02:51:39.968 INFO:tasks.rados.rados.0.smithi105.stdout:1171: expect deleted 2024-04-20T02:51:39.968 INFO:tasks.rados.rados.0.smithi105.stdout:1172: snap_remove snap 18 2024-04-20T02:51:39.994 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:39.994 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:40.006 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:40.044 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:40.044+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:40.134 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:40.134 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:40.145 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:40.146 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:40.145+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:40.209 INFO:tasks.rados.rados.0.smithi105.stdout:1172: done (6 left) 2024-04-20T02:51:40.209 INFO:tasks.rados.rados.0.smithi105.stdout:1173: write oid 36 current snap is 19 2024-04-20T02:51:40.209 INFO:tasks.rados.rados.0.smithi105.stdout:1166: got (16) Device or resource busy 2024-04-20T02:51:40.209 INFO:tasks.rados.rados.0.smithi105.stdout:check_oldest_snap_flushed oid 46 v 48 (ObjNum 192 snap 19 seq_num 192) unflushed 19 2024-04-20T02:51:40.209 INFO:tasks.rados.rados.0.smithi105.stdout:1168: expect (ObjNum 87 snap 2 seq_num 87) 2024-04-20T02:51:40.210 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:51:40.246 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:40.245+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:40.279 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:40.279 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:40.289 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:40.347 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:40.347+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:40.353 INFO:tasks.rados.rados.0.smithi105.stdout:1165: finishing write tid 1 to smithi10548823-8 2024-04-20T02:51:40.353 INFO:tasks.rados.rados.0.smithi105.stdout:1165: finishing write tid 2 to smithi10548823-8 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:1165: finishing write tid 3 to smithi10548823-8 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:1165: finishing write tid 4 to smithi10548823-8 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:1165: finishing write tid 5 to smithi10548823-8 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:1165: oid 8 updating version 0 to 62 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:1165: oid 8 version 62 is already newer than 61 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:1165: oid 8 updating version 62 to 63 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:1165: oid 8 version 63 is already newer than 60 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 63 (ObjNum 204 snap 19 seq_num 204) dirty exists 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:1165: left oid 8 (ObjNum 204 snap 19 seq_num 204) 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:1169: finishing write tid 1 to smithi10548823-24 2024-04-20T02:51:40.354 INFO:tasks.rados.rados.0.smithi105.stdout:1173: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-04-20T02:51:40.361 INFO:tasks.rados.rados.0.smithi105.stdout:1173: writing smithi10548823-36 from 679049 to 1397123 tid 1 2024-04-20T02:51:40.368 INFO:tasks.rados.rados.0.smithi105.stdout:1173: writing smithi10548823-36 from 2113302 to 2787690 tid 2 2024-04-20T02:51:40.373 INFO:tasks.rados.rados.0.smithi105.stdout:1173: writing smithi10548823-36 from 3335030 to 3479049 tid 3 2024-04-20T02:51:40.373 INFO:tasks.rados.rados.0.smithi105.stdout:1165: done (6 left) 2024-04-20T02:51:40.373 INFO:tasks.rados.rados.0.smithi105.stdout:1166: done (5 left) 2024-04-20T02:51:40.373 INFO:tasks.rados.rados.0.smithi105.stdout:1168: done (4 left) 2024-04-20T02:51:40.373 INFO:tasks.rados.rados.0.smithi105.stdout:1171: done (3 left) 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1174: write oid 44 current snap is 19 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1169: finishing write tid 2 to smithi10548823-24 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1169: finishing write tid 3 to smithi10548823-24 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1169: finishing write tid 4 to smithi10548823-24 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1169: finishing write tid 5 to smithi10548823-24 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1169: oid 24 updating version 0 to 201 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1169: oid 24 version 201 is already newer than 198 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1169: oid 24 version 201 is already newer than 200 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1169: oid 24 version 201 is already newer than 199 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 201 (ObjNum 205 snap 19 seq_num 205) dirty exists 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1169: left oid 24 (ObjNum 205 snap 19 seq_num 205) 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1170: finishing write tid 1 to smithi10548823-30 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1170: finishing write tid 2 to smithi10548823-30 2024-04-20T02:51:40.374 INFO:tasks.rados.rados.0.smithi105.stdout:1174: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-04-20T02:51:40.381 INFO:tasks.rados.rados.0.smithi105.stdout:1174: writing smithi10548823-44 from 695856 to 1489179 tid 1 2024-04-20T02:51:40.387 INFO:tasks.rados.rados.0.smithi105.stdout:1174: writing smithi10548823-44 from 2171784 to 2706183 tid 2 2024-04-20T02:51:40.392 INFO:tasks.rados.rados.0.smithi105.stdout:1174: writing smithi10548823-44 from 3362453 to 3495856 tid 3 2024-04-20T02:51:40.392 INFO:tasks.rados.rados.0.smithi105.stdout:1169: done (3 left) 2024-04-20T02:51:40.392 INFO:tasks.rados.rados.0.smithi105.stdout:1175: set_chunk oid 35 target oid 2024-04-20T02:51:40.426 INFO:tasks.rados.rados.0.smithi105.stdout:1175: set_chunk oid 35 offset: 541696 length: 1119744 target oid 79115d2b3eac251b782bf122bd3fdf212f27644fceb168c3ac12d46027557020 offset: 0 snap -1 2024-04-20T02:51:40.426 INFO:tasks.rados.rados.0.smithi105.stdout:1176: tier_promote oid 43 2024-04-20T02:51:40.426 INFO:tasks.rados.rados.0.smithi105.stdout:1170: finishing write tid 3 to smithi10548823-30 2024-04-20T02:51:40.426 INFO:tasks.rados.rados.0.smithi105.stdout:1170: finishing write tid 4 to smithi10548823-30 2024-04-20T02:51:40.427 INFO:tasks.rados.rados.0.smithi105.stdout:1170: finishing write tid 5 to smithi10548823-30 2024-04-20T02:51:40.427 INFO:tasks.rados.rados.0.smithi105.stdout:1170: oid 30 updating version 0 to 61 2024-04-20T02:51:40.427 INFO:tasks.rados.rados.0.smithi105.stdout:1170: oid 30 version 61 is already newer than 59 2024-04-20T02:51:40.427 INFO:tasks.rados.rados.0.smithi105.stdout:1170: oid 30 updating version 61 to 62 2024-04-20T02:51:40.427 INFO:tasks.rados.rados.0.smithi105.stdout:1170: oid 30 version 62 is already newer than 60 2024-04-20T02:51:40.427 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 62 (ObjNum 206 snap 19 seq_num 206) dirty exists 2024-04-20T02:51:40.427 INFO:tasks.rados.rados.0.smithi105.stdout:1170: left oid 30 (ObjNum 206 snap 19 seq_num 206) 2024-04-20T02:51:40.427 INFO:tasks.rados.rados.0.smithi105.stdout:1170: done (4 left) 2024-04-20T02:51:40.427 INFO:tasks.rados.rados.0.smithi105.stdout:1177: set_chunk oid 6 target oid 2024-04-20T02:51:40.448 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:40.447+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:40.452 INFO:tasks.rados.rados.0.smithi105.stdout:1177: set_chunk oid 6 offset: 1081344 length: 1765376 target oid 9bc0839fa29938fe093ba47e14aa201731c0ed08678e6240b2ef028681aed10e offset: 0 snap -1 2024-04-20T02:51:40.452 INFO:tasks.rados.rados.0.smithi105.stdout:1178: snap_remove snap 12 2024-04-20T02:51:40.474 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:40.475 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:40.485 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:40.549 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:40.550+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:40.650 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:40.650+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:40.654 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:40.654 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:40.664 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:40.751 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:40.751+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:40.833 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:40.833 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:40.843 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:40.852 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:40.852+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:40.952 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:40.953+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:40.972 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:40.972 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:40.982 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:41.054 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:41.054+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:41.110 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:41.111 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:41.121 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:41.125 INFO:tasks.rados.rados.0.smithi105.stdout:1178: done (5 left) 2024-04-20T02:51:41.126 INFO:tasks.rados.rados.0.smithi105.stdout:1179: set_chunk oid 14 target oid 2024-04-20T02:51:41.144 INFO:tasks.rados.rados.0.smithi105.stdout:1179: set_chunk oid 14 offset: 2885632 length: 476160 target oid 0cc15ac148102ed12ac32b547aa6787b37dc277b6e0d6da261d24bd29f84bf54 offset: 0 snap 17 2024-04-20T02:51:41.144 INFO:tasks.rados.rados.0.smithi105.stdout:1180: set_chunk oid 42 target oid 2024-04-20T02:51:41.154 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:41.154+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:41.162 INFO:tasks.rados.rados.0.smithi105.stdout:1180: set_chunk oid 42 offset: 90624 length: 680448 target oid a78a14781a139a3f0207fdc058facaee99e7b70aec6abd9942dd2062b8033819 offset: 0 snap -1 2024-04-20T02:51:41.162 INFO:tasks.rados.rados.0.smithi105.stdout:1181: delete oid 3 current snap is 19 2024-04-20T02:51:41.163 INFO:tasks.rados.rados.0.smithi105.stdout:1173: finishing write tid 1 to smithi10548823-36 2024-04-20T02:51:41.163 INFO:tasks.rados.rados.0.smithi105.stdout:1173: finishing write tid 2 to smithi10548823-36 2024-04-20T02:51:41.163 INFO:tasks.rados.rados.0.smithi105.stdout:1173: finishing write tid 3 to smithi10548823-36 2024-04-20T02:51:41.163 INFO:tasks.rados.rados.0.smithi105.stdout:1173: finishing write tid 4 to smithi10548823-36 2024-04-20T02:51:41.163 INFO:tasks.rados.rados.0.smithi105.stdout:1173: finishing write tid 5 to smithi10548823-36 2024-04-20T02:51:41.163 INFO:tasks.rados.rados.0.smithi105.stdout:1173: oid 36 updating version 0 to 207 2024-04-20T02:51:41.163 INFO:tasks.rados.rados.0.smithi105.stdout:1173: oid 36 version 207 is already newer than 205 2024-04-20T02:51:41.163 INFO:tasks.rados.rados.0.smithi105.stdout:1173: oid 36 version 207 is already newer than 206 2024-04-20T02:51:41.163 INFO:tasks.rados.rados.0.smithi105.stdout:1173: oid 36 version 207 is already newer than 202 2024-04-20T02:51:41.164 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 207 (ObjNum 207 snap 19 seq_num 207) dirty exists 2024-04-20T02:51:41.164 INFO:tasks.rados.rados.0.smithi105.stdout:1173: left oid 36 (ObjNum 207 snap 19 seq_num 207) 2024-04-20T02:51:41.164 INFO:tasks.rados.rados.0.smithi105.stdout:1175: finishing set_chunk to oid 35 2024-04-20T02:51:41.164 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 541696 length: 1119744 2024-04-20T02:51:41.164 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 124 snap 9 seq_num 124) dirty exists 2024-04-20T02:51:41.164 INFO:tasks.rados.rados.0.smithi105.stdout:1176: got (0) Success 2024-04-20T02:51:41.164 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 78 (ObjNum 187 snap 18 seq_num 187) dirty exists 2024-04-20T02:51:41.164 INFO:tasks.rados.rados.0.smithi105.stdout:1174: finishing write tid 1 to smithi10548823-44 2024-04-20T02:51:41.164 INFO:tasks.rados.rados.0.smithi105.stdout:1174: finishing write tid 2 to smithi10548823-44 2024-04-20T02:51:41.165 INFO:tasks.rados.rados.0.smithi105.stdout:1174: finishing write tid 3 to smithi10548823-44 2024-04-20T02:51:41.165 INFO:tasks.rados.rados.0.smithi105.stdout:1174: finishing write tid 4 to smithi10548823-44 2024-04-20T02:51:41.165 INFO:tasks.rados.rados.0.smithi105.stdout:1174: finishing write tid 5 to smithi10548823-44 2024-04-20T02:51:41.165 INFO:tasks.rados.rados.0.smithi105.stdout:1174: oid 44 updating version 0 to 72 2024-04-20T02:51:41.165 INFO:tasks.rados.rados.0.smithi105.stdout:1174: oid 44 version 72 is already newer than 69 2024-04-20T02:51:41.165 INFO:tasks.rados.rados.0.smithi105.stdout:1174: oid 44 version 72 is already newer than 70 2024-04-20T02:51:41.165 INFO:tasks.rados.rados.0.smithi105.stdout:1174: oid 44 version 72 is already newer than 71 2024-04-20T02:51:41.166 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 72 (ObjNum 208 snap 19 seq_num 208) dirty exists 2024-04-20T02:51:41.166 INFO:tasks.rados.rados.0.smithi105.stdout:1174: left oid 44 (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:51:41.166 INFO:tasks.rados.rados.0.smithi105.stdout:1177: finishing set_chunk to oid 6 2024-04-20T02:51:41.166 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 96 (ObjNum 194 snap 19 seq_num 194) dirty exists 2024-04-20T02:51:41.166 INFO:tasks.rados.rados.0.smithi105.stdout:1179: finishing set_chunk to oid 14 2024-04-20T02:51:41.167 INFO:tasks.rados.rados.0.smithi105.stdout:1173: done (7 left) 2024-04-20T02:51:41.167 INFO:tasks.rados.rados.0.smithi105.stdout:1174: done (6 left) 2024-04-20T02:51:41.167 INFO:tasks.rados.rados.0.smithi105.stdout:1175: done (5 left) 2024-04-20T02:51:41.167 INFO:tasks.rados.rados.0.smithi105.stdout:1176: done (4 left) 2024-04-20T02:51:41.167 INFO:tasks.rados.rados.0.smithi105.stdout:1177: done (3 left) 2024-04-20T02:51:41.167 INFO:tasks.rados.rados.0.smithi105.stdout:1179: done (2 left) 2024-04-20T02:51:41.167 INFO:tasks.rados.rados.0.smithi105.stdout:1181: done (1 left) 2024-04-20T02:51:41.167 INFO:tasks.rados.rados.0.smithi105.stdout:1182: read oid 4 snap -1 2024-04-20T02:51:41.167 INFO:tasks.rados.rados.0.smithi105.stdout:1182: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:51:41.167 INFO:tasks.rados.rados.0.smithi105.stdout:1183: snap_remove snap 9 2024-04-20T02:51:41.193 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:41.254 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:41.255+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:41.260 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:41.260 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:41.270 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:41.342 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:41.355 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:41.355+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:41.362 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:41.381 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:41.400 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:41.403 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:41.403 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:41.404 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:41.404 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:41.414 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:41.420 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:41.440 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:41.456 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:41.456+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:41.459 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:41.479 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:41.481 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:41.481 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:41.481 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:41.481 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:41.481 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:41.481 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:41.481 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:41.481 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:41.545 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:41.545 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:41.545 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:41.545 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:41.545 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:41.555 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:41.557 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:41.557+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:41.658 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:41.658+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:927 4.3 4:cf11ccc0:::smithi10548823-4:head [write 3121996~104915] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.307497525999999, 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058986447999999997, 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.686 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 927 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:51:41.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377548+0000", 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377607+0000", 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9413999999999997e-05 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377616+0000", 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4880000000000007e-06 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377621+0000", 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.682e-06 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407778+0000", 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030156110999999999 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407866+0000", 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8645000000000003e-05 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407985+0000", 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118874 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411741+0000", 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037563060000000001 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411854+0000", 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112898 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436438+0000", 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024583885 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436483+0000", 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5334999999999999e-05 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436535+0000", 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1780000000000002e-05 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:928 4.3 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (55) in=7b,truncate 3226911] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.307418920000003, 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059064512999999999, 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.692 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.693 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 928 2024-04-20T02:51:41.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.693 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:41.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377630+0000", 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7259999999999998e-06 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377633+0000", 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1489999999999998e-06 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377636+0000", 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7760000000000002e-06 2024-04-20T02:51:41.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410754+0000", 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033117944000000003 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410850+0000", 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6522999999999997e-05 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411008+0000", 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158157 2024-04-20T02:51:41.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436151+0000", 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025142813 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436160+0000", 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1460000000000002e-06 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436588+0000", 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00042776499999999998 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436631+0000", 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3084000000000003e-05 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436692+0000", 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0829000000000003e-05 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.698 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:929 4.3 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.307407779000002, 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059214058, 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 929 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377638+0000", 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377640+0000", 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5430000000000001e-06 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377643+0000", 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0900000000000001e-06 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377647+0000", 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2230000000000001e-06 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411271+0000", 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033623635999999998 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411329+0000", 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8043e-05 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436728+0000", 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025399300999999999 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436793+0000", 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5041000000000003e-05 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436853+0000", 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9446e-05 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.703 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:936 4.3 4:ca036f62:::smithi10548823-34:head [write 443718~615603] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:41.704 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:41.704 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.290372445999999, 2024-04-20T02:51:41.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060880228000000002, 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 936 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394672+0000", 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394825+0000", 2024-04-20T02:51:41.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015343299999999999 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394844+0000", 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8414999999999999e-05 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394859+0000", 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4738e-05 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411879+0000", 2024-04-20T02:51:41.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017020469999999999 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411954+0000", 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5239000000000002e-05 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412104+0000", 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014944399999999999 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436402+0000", 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024297828 2024-04-20T02:51:41.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436409+0000", 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1779999999999997e-06 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455390+0000", 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018981221 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455508+0000", 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117556 2024-04-20T02:51:41.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455554+0000", 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6570000000000003e-05 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:937 4.3 4:ca036f62:::smithi10548823-34:head [write 1564138~470341] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.285034115999999, 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060119655000000001, 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.710 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 937 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400010+0000", 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400222+0000", 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000211208 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400238+0000", 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419e-05 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400250+0000", 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2091e-05 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425851+0000", 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025600795999999999 2024-04-20T02:51:41.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425965+0000", 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011411799999999999 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426134+0000", 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168517 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436937+0000", 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01080328 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455351+0000", 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018414607 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460001+0000", 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046491809999999996 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460086+0000", 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5066000000000002e-05 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460132+0000", 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6174999999999998e-05 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:938 4.3 4:ca036f62:::smithi10548823-34:head [write 2633066~610652] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:41.716 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.278268660000002, 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054360654000000001, 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 938 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406775+0000", 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406947+0000", 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171619 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406962+0000", 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5248999999999999e-05 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406979+0000", 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7334999999999999e-05 2024-04-20T02:51:41.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436954+0000", 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029974218 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437007+0000", 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3553e-05 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437110+0000", 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455605+0000", 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018494295000000001 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455829+0000", 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022443100000000001 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461044+0000", 2024-04-20T02:51:41.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052150499999999997 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461090+0000", 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6100999999999998e-05 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461138+0000", 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8003000000000001e-05 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:939 4.3 4:ca036f62:::smithi10548823-34:head [setxattr _header (56) in=7b,truncate 3243718] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.278056349000003, 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054380381999999998, 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 939 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406989+0000", 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406993+0000", 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9400000000000004e-06 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406999+0000", 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4389999999999999e-06 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407008+0000", 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3190000000000001e-06 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.454982+0000", 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047973729999999999 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455051+0000", 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9197999999999997e-05 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455157+0000", 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106302 2024-04-20T02:51:41.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456120+0000", 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096236599999999998 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456272+0000", 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000152573 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461274+0000", 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005002055 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461318+0000", 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3282999999999999e-05 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461370+0000", 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2704999999999998e-05 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:940 4.3 4:ca036f62:::smithi10548823-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.278031800000001, 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054543401999999998, 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 940 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407014+0000", 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407017+0000", 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9790000000000002e-06 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407022+0000", 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5539999999999997e-06 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407027+0000", 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6099999999999999e-06 2024-04-20T02:51:41.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455643+0000", 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.048616331999999998 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455714+0000", 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0610000000000003e-05 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461403+0000", 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056888219999999996 2024-04-20T02:51:41.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461480+0000", 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7342000000000006e-05 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461558+0000", 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7780000000000004e-05 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.733 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.140386264999997, 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:51:41.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:51:41.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:51:41.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.737 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.752463804000001, 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:41.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:51:41.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:41.743 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.752203598000001, 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:51:41.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:51:41.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:51:41.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.752182130000001, 2024-04-20T02:51:41.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:41.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.558510198, 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:51:41.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.759 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:133 3.1 3:8efdf53d:::0188443c5b8894a8247535d52a547bece3a4c5f490fb220852b80e77d71ddcda:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.557785676000002, 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060974049000000002, 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:51:41.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127260+0000", 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127435+0000", 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017459800000000001 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127443+0000", 2024-04-20T02:51:41.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3790000000000006e-06 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127450+0000", 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5089999999999999e-06 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.129973+0000", 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025229639999999999 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130090+0000", 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117521 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130323+0000", 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023271199999999999 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.147581+0000", 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017258440999999999 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155339+0000", 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077577189999999997 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188139+0000", 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032800425000000001 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188195+0000", 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5773999999999998e-05 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188235+0000", 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9427e-05 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:136 3.1 3:a965252c:::a490375a0954219328244dfa32cf9967dfac8ed04667cb9b5ce15f4910db2ee6:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.557590458999996, 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060878320999999999, 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.765 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127455+0000", 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128197+0000", 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074176999999999999 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128208+0000", 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0498e-05 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128216+0000", 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2630000000000003e-06 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135797+0000", 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0075808669999999998 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135930+0000", 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013299000000000001 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.136192+0000", 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026213899999999999 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148053+0000", 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011860723 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155346+0000", 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0072930039999999996 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188270+0000", 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032924299999999997 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188298+0000", 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8041000000000001e-05 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188334+0000", 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6247e-05 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.770 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:139 3.1 3:90772484:::f9b8a83c02fab1c87f6aef9574ab68b1d6a75b657e100bac3693f85cac340cb2:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.550932533999998, 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054361675999999998, 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134113+0000", 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134645+0000", 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053281300000000004 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134662+0000", 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6503000000000001e-05 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134672+0000", 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8090000000000004e-06 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148073+0000", 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013401091 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148177+0000", 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104344 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148417+0000", 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023968899999999999 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155385+0000", 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0069679260000000002 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155725+0000", 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034046800000000002 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188383+0000", 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032658154000000002 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188433+0000", 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0059999999999998e-05 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188475+0000", 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2049999999999999e-05 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:140 3.1 3:b80bf598:::82930be3d6fd0d013eae1f4ab54a437986489fd06a4f4934c2432c14687252ab:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.550367279, 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053882434, 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134678+0000", 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134756+0000", 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7527999999999994e-05 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134763+0000", 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.092e-06 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0290000000000002e-06 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152310+0000", 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017540581999999999 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152431+0000", 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012166 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152621+0000", 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000189944 2024-04-20T02:51:41.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155992+0000", 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033703470000000001 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160642+0000", 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046505239999999996 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188508+0000", 2024-04-20T02:51:41.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027866134000000001 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188534+0000", 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5945999999999999e-05 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188561+0000", 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7300999999999998e-05 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.781 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1801 4.6 4:697da67d:::smithi10548823-48:head [setxattr _header (57) in=7b,truncate 2588278] snapc 13=[13,d] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.605267171000001, 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053362723000000001, 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1801 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079779+0000", 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079782+0000", 2024-04-20T02:51:41.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7840000000000001e-06 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079790+0000", 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3270000000000003e-06 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079799+0000", 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5040000000000008e-06 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123583+0000", 2024-04-20T02:51:41.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043783913000000001 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123714+0000", 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013130699999999999 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.123889+0000", 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017436200000000001 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132488+0000", 2024-04-20T02:51:41.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085989889999999996 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.132702+0000", 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021448600000000001 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133026+0000", 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032330500000000002 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133079+0000", 2024-04-20T02:51:41.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3127000000000001e-05 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133142+0000", 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3150000000000004e-05 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1802 4.6 4:697da67d:::smithi10548823-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.605233485999999, 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053694407999999999, 2024-04-20T02:51:41.787 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1802 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:51:41.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079812+0000", 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079815+0000", 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1329999999999999e-06 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079823+0000", 2024-04-20T02:51:41.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.401e-06 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079828+0000", 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4149999999999998e-06 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124387+0000", 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044558788000000002 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:41.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124468+0000", 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1447999999999997e-05 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133186+0000", 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0087175610000000004 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133344+0000", 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158393 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133507+0000", 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000162829 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2292 4.3 4:c07c6daa:::smithi10548823-7:head [write 544593~799729] snapc 1e=[1e,1d,1b,18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 7.5254670059999995, 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061161747000000002, 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:41.792 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2292 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:41.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159577+0000", 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160019+0000", 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000442674 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160045+0000", 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5446e-05 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:41.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160058+0000", 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3444e-05 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160487+0000", 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000428899 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160616+0000", 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012861099999999999 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:51:41.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160838+0000", 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000222415 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207115+0000", 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046276543000000003 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207164+0000", 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9497999999999999e-05 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220657+0000", 2024-04-20T02:51:41.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013492386 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220698+0000", 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1451e-05 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220741+0000", 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3078000000000003e-05 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:41.797 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:41.798 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:41.799 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:41.799 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:41.799+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:41.899 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:41.899+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:41.967 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:41.967 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:41.967 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:41.967 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:41.978 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:42.002 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:42.003+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:42.103 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:42.103+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:42.105 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:42.105 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:42.105 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:42.106 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:42.106 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:42.116 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:42.122 INFO:tasks.rados.rados.0.smithi105.stdout:1183: done (2 left) 2024-04-20T02:51:42.123 INFO:tasks.rados.rados.0.smithi105.stdout:1184: read oid 25 snap -1 2024-04-20T02:51:42.123 INFO:tasks.rados.rados.0.smithi105.stdout:1184: expect (ObjNum 169 snap 14 seq_num 169) 2024-04-20T02:51:42.123 INFO:tasks.rados.rados.0.smithi105.stdout:1185: set_chunk oid 39 target oid 2024-04-20T02:51:42.137 INFO:tasks.rados.rados.0.smithi105.stdout:1185: set_chunk oid 39 offset: 1748992 length: 164864 target oid 8c6084d2ac39c463905579a6bfaea8b4fee840b4eba4d235b40698db36de107a offset: 0 snap -1 2024-04-20T02:51:42.137 INFO:tasks.rados.rados.0.smithi105.stdout:1186: set_chunk oid 5 target oid 2024-04-20T02:51:42.137 INFO:tasks.rados.rados.0.smithi105.stdout:1186: set_chunk oid 5 offset: 558080 length: 2805760 target oid smithi10548823- offset: 558080 snap 17 2024-04-20T02:51:42.138 INFO:tasks.rados.rados.0.smithi105.stdout:1187: snap_remove snap 15 2024-04-20T02:51:42.206 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:42.206+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:42.223 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:51:42.265 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:42.265 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:42.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:42.265 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:42.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.265 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.678376686999997, 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:42.271 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.662358408999999, 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:51:42.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:51:42.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:51:42.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.662205180000001, 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:42.277 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.662095170999997, 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.662064794999999, 2024-04-20T02:51:42.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.657647376, 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:42.293 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:51:42.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:51:42.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:51:42.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:51:42.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:51:42.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:51:42.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.648395624999999, 2024-04-20T02:51:42.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:51:42.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:51:42.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:51:42.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.645873481999999, 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:51:42.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:51:42.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:51:42.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:42.313 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.645788615000001, 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:51:42.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.645771005, 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:51:42.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:51:42.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:991 4.b e32/22 4:d4308fef:::smithi10548823-6:head v 32'29, mlcod=30'27)", 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.538866751, 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046400134000000003, 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724649+0000", 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724907+0000", 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025779699999999998 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724924+0000", 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7110000000000001e-05 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724934+0000", 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5370000000000003e-06 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724984+0000", 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0791000000000002e-05 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.725011+0000", 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6299e-05 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771009+0000", 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045998144999999997 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771050+0000", 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1727999999999998e-05 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.296404397000003, 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058137664999999998, 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967112+0000", 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981895+0000", 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014783522 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981906+0000", 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0560000000000001e-05 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981912+0000", 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:42.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981956+0000", 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3455000000000003e-05 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981979+0000", 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3475000000000001e-05 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025222+0000", 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043242622000000001 2024-04-20T02:51:42.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025250+0000", 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8520000000000001e-05 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.874335598999998, 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045096171999999997, 2024-04-20T02:51:42.333 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401391+0000", 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012209697 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401410+0000", 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8754e-05 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401422+0000", 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2089999999999999e-05 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.403142+0000", 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0017199100000000001 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.403157+0000", 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.526e-05 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434233+0000", 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031076289999999999 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434278+0000", 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4478999999999999e-05 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:42.337 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.851396409000003, 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:51:42.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:51:42.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.845391620999997, 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.342 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:51:42.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:130 3.0 3:1b476ba0:::474a8d6c2b8981e0c13c883e0d254815cfd155e523c9b654ff4e5007d0a1fc09:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:42.346 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.136050613999998, 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045176642000000003, 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128171+0000", 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070576600000000003 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128183+0000", 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1542999999999999e-05 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128191+0000", 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5839999999999995e-06 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134592+0000", 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006400233 2024-04-20T02:51:42.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134784+0000", 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000191831 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135041+0000", 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000257123 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162360+0000", 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027319774000000002 2024-04-20T02:51:42.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162476+0000", 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115505 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172550+0000", 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01007366 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172598+0000", 2024-04-20T02:51:42.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8216000000000001e-05 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172643+0000", 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5305000000000001e-05 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:135 3.0 3:12659911:::ad802ca609ac9a6b9a438bdacacdb4fb27f17cd228f3ff5506eb31f643d749ff:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:42.352 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.131022657999999, 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045928208999999998, 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132493+0000", 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133291+0000", 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079715799999999996 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133301+0000", 2024-04-20T02:51:42.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0533000000000001e-05 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133309+0000", 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9910000000000002e-06 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151891+0000", 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018582155 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152003+0000", 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112064 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152211+0000", 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020773100000000001 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162492+0000", 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010281047999999999 2024-04-20T02:51:42.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169484+0000", 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006991929 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178312+0000", 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0088282789999999996 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178374+0000", 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2123000000000001e-05 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178423+0000", 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201999999999997e-05 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.358 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:138 3.0 3:2a051c65:::f4051f504bf0c69c0b0fcc37af36dc6836e32eef2351291df998149766aaa9b9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.130201241000002, 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045257957000000001, 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:42.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133315+0000", 2024-04-20T02:51:42.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133606+0000", 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029134199999999999 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133615+0000", 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2479999999999996e-06 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133621+0000", 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7739999999999999e-06 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159641+0000", 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026020008000000001 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159813+0000", 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017143200000000001 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:42.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160027+0000", 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021374400000000001 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.163217+0000", 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031905869999999999 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169517+0000", 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063002290000000001 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178484+0000", 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0089668070000000002 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178534+0000", 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9654000000000003e-05 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178574+0000", 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9929999999999999e-05 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.364 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.174883490999999, 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049605908999999997, 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088632+0000", 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090096+0000", 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001463675 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090112+0000", 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5557999999999999e-05 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090125+0000", 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3794999999999999e-05 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090196+0000", 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0537000000000006e-05 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090224+0000", 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8555999999999999e-05 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138207+0000", 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047982536999999999 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138240+0000", 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2480000000000001e-05 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.368 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.1221754980000007, 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056137746000000002, 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141340+0000", 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142751+0000", 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014113540000000001 2024-04-20T02:51:42.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142770+0000", 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8450000000000001e-05 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142780+0000", 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0284e-05 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142982+0000", 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000202244 2024-04-20T02:51:42.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.143007+0000", 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4929000000000001e-05 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197446+0000", 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054438519999999997 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197479+0000", 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3597999999999999e-05 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.372 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.373 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.373 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.373 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.373 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:42.374 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:42.375 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:42.374+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:42.475 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:42.475+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:42.546 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:42.546 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:42.546 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:42.546 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:42.556 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:42.597 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:42.597+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:42.678 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:42.678+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:42.723 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:42.723 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:42.723 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:42.723 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:42.723 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:42.733 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:42.779 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:42.779+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:42.862 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.916977975000002, 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041399200999999997, 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945227+0000", 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:42.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947429+0000", 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022028989999999999 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947448+0000", 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.821e-05 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947460+0000", 2024-04-20T02:51:42.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2373e-05 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947511+0000", 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0844999999999999e-05 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947531+0000", 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0017000000000002e-05 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986580+0000", 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039049213999999999 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986627+0000", 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7070000000000002e-05 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1145 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'30, mlcod=34'26)", 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:42.867 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.898247275999999, 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026752594000000001, 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963958+0000", 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964662+0000", 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070467299999999995 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964678+0000", 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5597999999999999e-05 2024-04-20T02:51:42.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964687+0000", 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2979999999999997e-06 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968184+0000", 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034972250000000001 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968211+0000", 2024-04-20T02:51:42.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6829999999999999e-05 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990640+0000", 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022429317000000001 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990711+0000", 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0794e-05 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:129 3.0 e42/21 3:0256f537:::3f82a762bf2204de901e10565f70e759ee2a26bf55ffef4c8d5027a47e96ec31:head v 42'78, mlcod=42'76)", 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.727558701, 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024418002000000001, 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134646+0000", 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012265099999999999 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134789+0000", 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9924000000000001e-05 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134799+0000", 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0042e-05 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134842+0000", 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3791000000000001e-05 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134866+0000", 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3791e-05 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159017+0000", 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024150919 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159065+0000", 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201000000000001e-05 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.875 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:130 3.0 e42/21 3:1b476ba0:::474a8d6c2b8981e0c13c883e0d254815cfd155e523c9b654ff4e5007d0a1fc09:head v 42'79, mlcod=42'76)", 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.727029088999998, 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023928118000000002, 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135176+0000", 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135295+0000", 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118847 2024-04-20T02:51:42.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135304+0000", 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4280000000000002e-06 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135311+0000", 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4810000000000003e-06 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141973+0000", 2024-04-20T02:51:42.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066622219999999998 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.142009+0000", 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5725000000000001e-05 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159073+0000", 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017064019999999999 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159105+0000", 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2081e-05 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:134 3.0 e42/21 3:33aaac3f:::d0aca4dab23a9f22fb58211e350c7c1c1b7601bafbb496b93afbbc91e2ed921c:head v 42'81, mlcod=42'76)", 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.717536502000002, 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024591152000000002, 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144949+0000", 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028058500000000002 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144959+0000", 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0335e-05 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144967+0000", 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6040000000000004e-06 2024-04-20T02:51:42.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151637+0000", 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066694670000000001 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151659+0000", 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2855000000000001e-05 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169224+0000", 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017565041 2024-04-20T02:51:42.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169261+0000", 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6179e-05 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1462 3.0 e44/21 3:29b349f1:::ea1112aea5e7be379214f0429a6e898cf4f4a05bdbb29b6cd15764b4776418b1:head v 44'85, mlcod=42'83)", 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.684416616999997, 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034060275000000001, 2024-04-20T02:51:42.884 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178932+0000", 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011426069999999999 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178944+0000", 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.287e-05 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178962+0000", 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7669999999999999e-05 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179013+0000", 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1295000000000003e-05 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179038+0000", 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4859999999999999e-05 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211813+0000", 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032775190000000003 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211850+0000", 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6216e-05 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1629 3.0 e51/21 3:088cc2f3:::a1744cc895df5e9d3e624b281fcb9edc57f3f342d1f108d4c38dc778f219a41a:head v 51'93, mlcod=45'91)", 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:42.888 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.059117280000002, 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035323188999999998, 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803088+0000", 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804187+0000", 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00109891 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804201+0000", 2024-04-20T02:51:42.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3772e-05 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804213+0000", 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2335e-05 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804285+0000", 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1271000000000003e-05 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804306+0000", 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1359999999999999e-05 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838377+0000", 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034070994 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838412+0000", 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4790999999999999e-05 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1653 4.6 e51/22 4:697da67d:::smithi10548823-48:head v 51'41, mlcod=45'37)", 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.437378172000003, 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037653640000000002, 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:42.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424827+0000", 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425066+0000", 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023957399999999999 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425117+0000", 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1082999999999999e-05 2024-04-20T02:51:42.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425146+0000", 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8129000000000001e-05 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425217+0000", 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1402000000000005e-05 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425278+0000", 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1498e-05 2024-04-20T02:51:42.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462456+0000", 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037177106000000001 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462481+0000", 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5851e-05 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1786 4.d e56/22 4:b376c20e:::smithi10548823-49:head v 56'80, mlcod=55'77)", 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.050286634999999, 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028136330000000001, 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812644+0000", 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072510100000000002 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812656+0000", 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1688999999999999e-05 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812663+0000", 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812707+0000", 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4573999999999997e-05 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812722+0000", 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5319e-05 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840021+0000", 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027298111999999999 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840056+0000", 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5086000000000003e-05 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1799 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'53, mlcod=54'49)", 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.787957614, 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050527214000000001, 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.902 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:42.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074614+0000", 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036631499999999998 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074624+0000", 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8330000000000006e-06 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074632+0000", 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3599999999999998e-06 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074683+0000", 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1137999999999997e-05 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074704+0000", 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1616000000000001e-05 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124714+0000", 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050009642 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124776+0000", 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1581000000000006e-05 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.773461518000001, 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046184680999999998, 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:42.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088743+0000", 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090327+0000", 2024-04-20T02:51:42.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001584071 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090343+0000", 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5193e-05 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090354+0000", 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1336e-05 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090411+0000", 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6589999999999999e-05 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090431+0000", 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0579000000000001e-05 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134883+0000", 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044451877000000001 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134929+0000", 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6131000000000003e-05 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1800 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'54, mlcod=54'49)", 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.761055444, 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027029681999999999, 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.911 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101150+0000", 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101284+0000", 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000134688 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101295+0000", 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101305+0000", 2024-04-20T02:51:42.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0930999999999999e-05 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101347+0000", 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1522e-05 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101369+0000", 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2027000000000001e-05 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128138+0000", 2024-04-20T02:51:42.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026768626 2024-04-20T02:51:42.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128180+0000", 2024-04-20T02:51:42.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2568999999999998e-05 2024-04-20T02:51:42.915 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.915 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1807 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'87, mlcod=56'82)", 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.753374617999999, 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026347280000000001, 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:42.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108830+0000", 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109221+0000", 2024-04-20T02:51:42.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039087699999999998 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109239+0000", 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8904000000000001e-05 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109271+0000", 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1319000000000003e-05 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120423+0000", 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011151912999999999 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120454+0000", 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1464999999999998e-05 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135158+0000", 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014703554000000001 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135179+0000", 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0965999999999999e-05 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2038 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'159, mlcod=0'0)", 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.280586045, 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044306238999999997, 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581964+0000", 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034420099999999998 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581980+0000", 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6578999999999998e-05 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581991+0000", 2024-04-20T02:51:42.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0949e-05 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582734+0000", 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074277299999999996 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582761+0000", 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6627000000000001e-05 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625882+0000", 2024-04-20T02:51:42.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043121617000000001 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625926+0000", 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3838999999999998e-05 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2039 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'160, mlcod=0'0)", 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.26018796, 2024-04-20T02:51:42.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027318719000000002, 2024-04-20T02:51:42.925 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.925 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.925 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:42.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602017+0000", 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602364+0000", 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000347139 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602375+0000", 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.128e-05 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602384+0000", 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2600000000000005e-06 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605223+0000", 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0028393400000000001 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605245+0000", 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2078000000000001e-05 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629297+0000", 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024051605 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629337+0000", 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0126000000000002e-05 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2137 3.0 e70/21 3:0d26903d:::7146376437549e0b99264d159bf1e9c8bc43ac1c7eb519f65899cfd592a43a4a:head v 70'98, mlcod=55'96)", 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 15.078059976, 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024394852000000002, 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.929 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784144+0000", 2024-04-20T02:51:42.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785280+0000", 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001135741 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785299+0000", 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8845999999999999e-05 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785310+0000", 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1450999999999999e-05 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785362+0000", 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1798000000000001e-05 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785388+0000", 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5763999999999999e-05 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808507+0000", 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023118742000000001 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808541+0000", 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4258000000000003e-05 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2142 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'65, mlcod=66'61)", 2024-04-20T02:51:42.933 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 15.045360562999999, 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033365672999999998, 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:42.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816981+0000", 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135952 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816994+0000", 2024-04-20T02:51:42.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3405e-05 2024-04-20T02:51:42.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817003+0000", 2024-04-20T02:51:42.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4319999999999999e-06 2024-04-20T02:51:42.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817055+0000", 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1842999999999998e-05 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817076+0000", 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1375000000000001e-05 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850179+0000", 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033103051000000001 2024-04-20T02:51:42.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850211+0000", 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1896000000000001e-05 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2143 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'66, mlcod=66'61)", 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 15.031494721, 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023949208999999999, 2024-04-20T02:51:42.938 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830710+0000", 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830883+0000", 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017331000000000001 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830903+0000", 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0043000000000001e-05 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830914+0000", 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1263999999999999e-05 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.836558+0000", 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056437309999999999 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.836587+0000", 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:51:42.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.854633+0000", 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01804584 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.854660+0000", 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7708e-05 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.942 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.7208259479999999, 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039421115999999999, 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141379+0000", 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142797+0000", 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001418702 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142816+0000", 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8907999999999999e-05 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142825+0000", 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1020000000000003e-06 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142872+0000", 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6168999999999998e-05 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142891+0000", 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9561000000000001e-05 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180704+0000", 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037812643 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180801+0000", 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7361999999999994e-05 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.946 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2280 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'105, mlcod=73'100)", 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.7001446710000003, 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029657706999999998, 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:42.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162060+0000", 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162455+0000", 2024-04-20T02:51:42.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039569799999999999 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162472+0000", 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6820000000000002e-05 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162485+0000", 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2388999999999999e-05 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163758+0000", 2024-04-20T02:51:42.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001273906 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163784+0000", 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5352000000000001e-05 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191663+0000", 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027879583999999999 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191719+0000", 2024-04-20T02:51:42.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640000000000003e-05 2024-04-20T02:51:42.951 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.951 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.951 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.951 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:42.951 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:42.951 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:42.952 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:42.953 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:42.952+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:43.053 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:43.053+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:43.081 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:43.081 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:43.081 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:43.081 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:43.091 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:43.126 INFO:tasks.rados.rados.0.smithi105.stdout:1187: done (5 left) 2024-04-20T02:51:43.126 INFO:tasks.rados.rados.0.smithi105.stdout:1188: read oid 10 snap -1 2024-04-20T02:51:43.127 INFO:tasks.rados.rados.0.smithi105.stdout:1188: expect (ObjNum 156 snap 10 seq_num 156) 2024-04-20T02:51:43.127 INFO:tasks.rados.rados.0.smithi105.stdout:1189: read oid 49 snap 17 2024-04-20T02:51:43.127 INFO:tasks.rados.rados.0.smithi105.stdout:1189: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:43.127 INFO:tasks.rados.rados.0.smithi105.stdout:1190: read oid 50 snap -1 2024-04-20T02:51:43.128 INFO:tasks.rados.rados.0.smithi105.stdout:1190: expect (ObjNum 146 snap 10 seq_num 146) 2024-04-20T02:51:43.128 INFO:tasks.rados.rados.0.smithi105.stdout:1191: rollback oid 2 current snap is 19 2024-04-20T02:51:43.128 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 2 to 17 2024-04-20T02:51:43.128 INFO:tasks.rados.rados.0.smithi105.stdout:1180: finishing set_chunk to oid 42 2024-04-20T02:51:43.128 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 21 (ObjNum 108 snap 6 seq_num 108) dirty exists 2024-04-20T02:51:43.128 INFO:tasks.rados.rados.0.smithi105.stdout:1180: done (8 left) 2024-04-20T02:51:43.128 INFO:tasks.rados.rados.0.smithi105.stdout:1192: tier_evict oid 6 2024-04-20T02:51:43.128 INFO:tasks.rados.rados.0.smithi105.stdout:1192: tier_evict oid 6 snap -1 2024-04-20T02:51:43.128 INFO:tasks.rados.rados.0.smithi105.stdout:1182: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:51:43.141 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:51:43.142 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:51:43.142 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-20T02:51:43.142 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-20T02:51:43.143 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:51:43.153 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:43.153+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:43.255 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:43.255+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:43.274 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:43.335 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:43.335 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:43.335 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:43.335 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:43.336 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:43.356 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:43.356+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:43.394 INFO:tasks.rados.rados.0.smithi105.stdout:1184: expect (ObjNum 169 snap 14 seq_num 169) 2024-04-20T02:51:43.457 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:43.456+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:43.492 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:43.504 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:51:43.504 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:43.504 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 86, 2024-04-20T02:51:43.504 INFO:teuthology.orchestra.run.smithi105.stdout: "fsid": "a524a365-eaec-4ebd-aa4d-0d76d55a252f", 2024-04-20T02:51:43.504 INFO:teuthology.orchestra.run.smithi105.stdout: "created": "2024-04-20T02:50:07.356902+0000", 2024-04-20T02:51:43.504 INFO:teuthology.orchestra.run.smithi105.stdout: "modified": "2024-04-20T02:51:43.119988+0000", 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "last_up_change": "2024-04-20T02:50:54.351311+0000", 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "last_in_change": "2024-04-20T02:51:00.386783+0000", 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_num": 5799936, 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_set": [ 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "pglog_hardlimit", 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "purged_snapdirs", 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_deletes", 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "sortbitwise" 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_version": 5, 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T02:51:43.505 INFO:teuthology.orchestra.run.smithi105.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_snapshot": "", 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_max": 4, 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "max_osd": 8, 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "require_min_compat_client": "luminous", 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "min_compat_client": "jewel", 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "require_osd_release": "reef", 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_crimson": false, 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "pools": [ 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 1, 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": ".mgr", 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:17.498569+0000", 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-20T02:51:43.506 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool", 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 1, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 1, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 1, 2024-04-20T02:51:43.507 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 1, 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 1, 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "0.0", 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 0, 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 0, 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 0, 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "0'0", 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "0'0" 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "17", 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:51:43.508 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 0, 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 0, 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:51:43.509 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:51:43.510 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "options": { 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_max": 32, 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_min": 1 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": { 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr": {} 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 7.8899998664855957, 2024-04-20T02:51:43.511 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 7.8899998664855957, 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 0.37999999523162842, 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 3, 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 3, 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 2, 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": "rbd", 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:25.955448+0000", 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool", 2024-04-20T02:51:43.512 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 8, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 8, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 8, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 8, 2024-04-20T02:51:43.513 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 8, 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "0.0", 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 0, 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 0, 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 0, 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "0'0", 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "0'0" 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "20", 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:51:43.514 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 0, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 0, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:51:43.515 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:51:43.516 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "options": {}, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": { 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd": {} 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 2, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 2, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 1, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 2, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 2, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:51:43.517 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 3, 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": "low_tier", 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:33.168857+0000", 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool", 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:51:43.518 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 4, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 4, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 4, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 4, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 4, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "0.0", 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 0, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 0, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 0, 2024-04-20T02:51:43.519 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "0'0", 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "0'0" 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "23", 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 0, 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 0, 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:51:43.520 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:51:43.521 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:51:43.522 INFO:teuthology.orchestra.run.smithi105.stdout: "options": {}, 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": {}, 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 1.9900000095367432, 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 1.9900000095367432, 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 0.87999999523162842, 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 1.75, 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 1.75, 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.523 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 4, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": "unique_pool_0", 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:33.714259+0000", 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 8193, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:51:43.524 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 24, 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 16, 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 16, 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 16, 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 24, 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "4.18", 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 85, 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 84, 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 84, 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "82'63", 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "48'47" 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "86", 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:51:43.525 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "85", 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "83", 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 38, 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 86, 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:51:43.526 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:51:43.527 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "options": { 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "dedup_cdc_chunk_size": 131072, 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "dedup_tier": 3, 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "fingerprint_algorithm": "sha256" 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": { 2024-04-20T02:51:43.528 INFO:teuthology.orchestra.run.smithi105.stdout: "rados": {} 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 2.3299999237060547, 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 2.3299999237060547, 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 1, 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: "osds": [ 2024-04-20T02:51:43.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.530 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 0, 2024-04-20T02:51:43.530 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6", 2024-04-20T02:51:43.530 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:51:43.530 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:51:43.530 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:51:43.530 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:51:43.530 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:51:43.530 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:51:43.530 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 15, 2024-04-20T02:51:43.530 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 73, 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6812", 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.531 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6813", 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6815", 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.532 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6814", 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6812/43330", 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6813/43330", 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6815/43330", 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6814/43330", 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:51:43.533 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 1, 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "c5ae8a64-fc7c-47ab-af6a-4b762a840645", 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 13, 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 72, 2024-04-20T02:51:43.534 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6804", 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.535 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6805", 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6807", 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:51:43.536 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6806", 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6804/43331", 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6805/43331", 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6807/43331", 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6806/43331", 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:51:43.537 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 2, 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "336e852e-5fa6-4ac0-b053-d31ad40d7c4e", 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 13, 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 27, 2024-04-20T02:51:43.538 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6800", 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43326 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.539 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6801", 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43326 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6803", 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43326 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.540 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6802", 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43326 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6800/43326", 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6801/43326", 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6803/43326", 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6802/43326", 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:51:43.541 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 3, 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "695cab3a-74d8-4046-ac61-2656dc0d893e", 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 13, 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 73, 2024-04-20T02:51:43.542 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6808", 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.543 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6809", 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6811", 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:51:43.544 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6810", 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6808/43329", 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6809/43329", 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6811/43329", 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6810/43329", 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:51:43.545 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 4, 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "1d41da44-d894-40ac-acaa-aa0df15296c6", 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 0.9784088134765625, 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 15, 2024-04-20T02:51:43.546 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 81, 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6808", 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.547 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6809", 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6811", 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.548 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6810", 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6808/43330", 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6809/43330", 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6811/43330", 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6810/43330", 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:51:43.549 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 5, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "3e1c33fa-6c8c-4898-99bc-462ecef25f30", 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 15, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 25, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 38, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 85, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 26, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:51:43.550 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6802", 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6804", 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:51:43.551 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6807", 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.552 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6806", 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6802/52590", 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6804/52590", 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6807/52590", 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6806/52590", 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:51:43.553 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 6, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "fd1fc124-7cf8-4628-9076-1f27d6385676", 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 13, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 74, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:51:43.554 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6800", 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6801", 2024-04-20T02:51:43.555 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6805", 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:51:43.556 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6803", 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6800/43329", 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6801/43329", 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6805/43329", 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6803/43329", 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 7, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "53ae8193-5c3e-4698-aacd-df2ad1db9143", 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 15, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 27, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:51:43.558 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6812", 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6813", 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:51:43.559 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6815", 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:51:43.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6814", 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6812/43351", 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6813/43351", 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6815/43351", 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6814/43351", 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.561 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_xinfo": [ 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 0, 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:51:25.390868+0000", 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.562 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 1, 2024-04-20T02:51:43.563 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:51:43.563 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:51:43.563 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:51:43.563 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:51:43.563 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:51:43.563 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:51:21.956838+0000", 2024-04-20T02:51:43.563 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:51:43.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.563 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 2, 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:51:37.987332+0000", 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 3, 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:51:43.564 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:51:33.801631+0000", 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 4, 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:51:25.375647+0000", 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 5, 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:51:21.239839+0000", 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 26 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 6, 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:51:43.566 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:51:16.692530+0000", 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 7, 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:51:20.543600+0000", 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:51:43.567 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_upmap": [], 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_upmap_items": [], 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_upmap_primaries": [], 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_temp": [ 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: "pgid": "4.16", 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: "osds": [ 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: 0, 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: 3, 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: 7 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.568 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_temp": [], 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "blocklist": {}, 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "range_blocklist": {}, 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profiles": { 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "default": { 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "crush-failure-domain": "osd", 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "k": "2", 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "m": "1", 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "plugin": "jerasure", 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "technique": "reed_sol_van" 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps_queue": [ 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.569 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 4, 2024-04-20T02:51:43.570 INFO:teuthology.orchestra.run.smithi105.stdout: "snaps": [ 2024-04-20T02:51:43.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.570 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 13, 2024-04-20T02:51:43.570 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:51:43.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.571 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 19, 2024-04-20T02:51:43.571 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:51:43.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 21, 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 2 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 24, 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 27, 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 2 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 30, 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: "new_removed_snaps": [ 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 4, 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: "snaps": [ 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 27, 2024-04-20T02:51:43.573 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: "new_purged_snaps": [], 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_node_flags": {}, 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: "device_class_flags": {}, 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_mode": { 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_mode_enabled": false, 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_bucket_count": 0, 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: "degraded_stretch_mode": 0, 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: "recovering_stretch_mode": 0, 2024-04-20T02:51:43.574 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_mode_bucket": 0 2024-04-20T02:51:43.575 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.575 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:43.576 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:43.576 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.966787267999997, 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.577 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:51:43.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:51:43.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:51:43.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:51:43.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:51:43.583 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.583 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.583 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.960566194000002, 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.584 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:51:43.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:51:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:51:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:51:43.589 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.952165989999997, 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:51:43.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:51:43.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:51:43.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:51:43.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:43.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:51:43.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:51:43.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:51:43.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:51:43.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:51:43.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:51:43.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:51:43.596 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.596 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.596 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:971 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (56) in=7b,truncate 3277332] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.951847921000002, 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051333776999999997, 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 971 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558678+0000", 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:43.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558686+0000", 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7330000000000003e-06 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558693+0000", 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.866e-06 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558698+0000", 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.49e-06 2024-04-20T02:51:43.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594685+0000", 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035986753000000003 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594792+0000", 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107129 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594958+0000", 2024-04-20T02:51:43.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000165429 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595721+0000", 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076360100000000004 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.596242+0000", 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052083800000000003 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609915+0000", 2024-04-20T02:51:43.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013673431 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609958+0000", 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2759000000000002e-05 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610013+0000", 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4421e-05 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.951823345000001, 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:51:43.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.145501342000003, 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:51:43.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:51:43.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.613 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.138238379999997, 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.614 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:51:43.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.132295851000002, 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.620 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:43.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:51:43.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:51:43.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:43.626 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.132092858999997, 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:43.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:51:43.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:51:43.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:51:43.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.132069932, 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:43.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.640 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.121986198000002, 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:43.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:51:43.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:51:43.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:51:43.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.105100790000002, 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.088824348999999, 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:51:43.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.064268877000003, 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:51:43.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:51:43.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:51:43.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:51:43.659 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.749953100999999, 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.660 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2046 4.11 4:8b927dd6:::smithi10548823-28:head [write 2025476~459397] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.767100+0000", 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.743427066999999, 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050129193000000002, 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.666 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.667 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.667 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.667 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2046 2024-04-20T02:51:43.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.667 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767100+0000", 2024-04-20T02:51:43.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767100+0000", 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767098+0000", 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767266+0000", 2024-04-20T02:51:43.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000167901 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767281+0000", 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5478999999999999e-05 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767294+0000", 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3259e-05 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.786324+0000", 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019029404 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.786421+0000", 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6897000000000004e-05 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.786579+0000", 2024-04-20T02:51:43.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015888299999999999 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804648+0000", 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018068208999999998 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804685+0000", 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7832e-05 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817139+0000", 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012453862 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817184+0000", 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4437000000000002e-05 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817229+0000", 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5136000000000003e-05 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.672 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2047 4.11 4:8b927dd6:::smithi10548823-28:head [write 2971091~70976] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.770122+0000", 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.740404713, 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.048007332999999999, 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2047 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.673 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770122+0000", 2024-04-20T02:51:43.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770122+0000", 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770120+0000", 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770185+0000", 2024-04-20T02:51:43.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4204999999999999e-05 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770199+0000", 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4362000000000001e-05 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770212+0000", 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2649999999999999e-05 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804711+0000", 2024-04-20T02:51:43.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034499582000000001 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804818+0000", 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107203 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804983+0000", 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016515200000000001 2024-04-20T02:51:43.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817237+0000", 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012254034 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817905+0000", 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00066738500000000005 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818036+0000", 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013161999999999999 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818081+0000", 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4734999999999998e-05 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818129+0000", 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8167999999999997e-05 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.679 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2048 4.11 4:8b927dd6:::smithi10548823-28:head [setxattr _header (57) in=7b,truncate 3042067] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.770221+0000", 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.740306111999999, 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049037753000000003, 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2048 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770221+0000", 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770221+0000", 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770220+0000", 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770223+0000", 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.32e-06 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770230+0000", 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2759999999999997e-06 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770238+0000", 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2539999999999992e-06 2024-04-20T02:51:43.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816299+0000", 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046060940000000002 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816426+0000", 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012759199999999999 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816612+0000", 2024-04-20T02:51:43.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000185885 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818165+0000", 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001552138 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818192+0000", 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7271e-05 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819143+0000", 2024-04-20T02:51:43.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00095093499999999997 2024-04-20T02:51:43.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819199+0000", 2024-04-20T02:51:43.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6334e-05 2024-04-20T02:51:43.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819258+0000", 2024-04-20T02:51:43.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9373000000000002e-05 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2049 4.11 4:8b927dd6:::smithi10548823-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.740284326000001, 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049230470999999998, 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:43.686 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2049 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770245+0000", 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4430000000000002e-06 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770251+0000", 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2929999999999997e-06 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770260+0000", 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2930000000000006e-06 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816910+0000", 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046650129999999998 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817004+0000", 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3665999999999994e-05 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819289+0000", 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002284847 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819379+0000", 2024-04-20T02:51:43.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0274999999999999e-05 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819473+0000", 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3882999999999995e-05 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2050 4.0 4:007b2a87:::smithi10548823-39:head [write 658874~771437,stat] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.778080+0000", 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.732447253, 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051190877000000003, 2024-04-20T02:51:43.691 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2050 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778080+0000", 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778080+0000", 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778077+0000", 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778477+0000", 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039974900000000002 2024-04-20T02:51:43.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778496+0000", 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9335e-05 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778510+0000", 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3377000000000001e-05 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778572+0000", 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1898999999999998e-05 2024-04-20T02:51:43.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778685+0000", 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011372800000000001 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778917+0000", 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195799999999999 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817646+0000", 2024-04-20T02:51:43.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038729087000000002 2024-04-20T02:51:43.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:43.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817828+0000", 2024-04-20T02:51:43.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181986 2024-04-20T02:51:43.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:43.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.829157+0000", 2024-04-20T02:51:43.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011328379 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.829216+0000", 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8962000000000002e-05 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.829270+0000", 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4727e-05 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.697 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:43.698 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:43.698 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:43.699 INFO:tasks.thrashosds.thrasher:j is {'epoch': 86, 'fsid': 'a524a365-eaec-4ebd-aa4d-0d76d55a252f', 'created': '2024-04-20T02:50:07.356902+0000', 'modified': '2024-04-20T02:51:43.119988+0000', 'last_up_change': '2024-04-20T02:50:54.351311+0000', 'last_in_change': '2024-04-20T02:51:00.386783+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T02:50:17.498569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-20T02:50:25.955448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-04-20T02:50:33.168857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-20T02:50:33.714259+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 24, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 24, 'last_pg_merge_meta': {'source_pgid': '4.18', 'ready_epoch': 85, 'last_epoch_started': 84, 'last_epoch_clean': 84, 'source_version': "82'63", 'target_version': "48'47"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '85', 'last_force_op_resend_preluminous': '83', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 38, 'snap_epoch': 86, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6812', 'nonce': 43330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6813', 'nonce': 43330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6815', 'nonce': 43330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6814', 'nonce': 43330}]}, 'public_addr': '172.21.15.105:6812/43330', 'cluster_addr': '172.21.15.105:6813/43330', 'heartbeat_back_addr': '172.21.15.105:6815/43330', 'heartbeat_front_addr': '172.21.15.105:6814/43330', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5ae8a64-fc7c-47ab-af6a-4b762a840645', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6804', 'nonce': 43331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6805', 'nonce': 43331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6807', 'nonce': 43331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6806', 'nonce': 43331}]}, 'public_addr': '172.21.15.105:6804/43331', 'cluster_addr': '172.21.15.105:6805/43331', 'heartbeat_back_addr': '172.21.15.105:6807/43331', 'heartbeat_front_addr': '172.21.15.105:6806/43331', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '336e852e-5fa6-4ac0-b053-d31ad40d7c4e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6800', 'nonce': 43326}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6801', 'nonce': 43326}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6803', 'nonce': 43326}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6802', 'nonce': 43326}]}, 'public_addr': '172.21.15.105:6800/43326', 'cluster_addr': '172.21.15.105:6801/43326', 'heartbeat_back_addr': '172.21.15.105:6803/43326', 'heartbeat_front_addr': '172.21.15.105:6802/43326', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '695cab3a-74d8-4046-ac61-2656dc0d893e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6808', 'nonce': 43329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6809', 'nonce': 43329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6811', 'nonce': 43329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6810', 'nonce': 43329}]}, 'public_addr': '172.21.15.105:6808/43329', 'cluster_addr': '172.21.15.105:6809/43329', 'heartbeat_back_addr': '172.21.15.105:6811/43329', 'heartbeat_front_addr': '172.21.15.105:6810/43329', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1d41da44-d894-40ac-acaa-aa0df15296c6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9784088134765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6808', 'nonce': 43330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6809', 'nonce': 43330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6811', 'nonce': 43330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6810', 'nonce': 43330}]}, 'public_addr': '172.21.15.138:6808/43330', 'cluster_addr': '172.21.15.138:6809/43330', 'heartbeat_back_addr': '172.21.15.138:6811/43330', 'heartbeat_front_addr': '172.21.15.138:6810/43330', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '3e1c33fa-6c8c-4898-99bc-462ecef25f30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 15, 'last_clean_end': 25, 'up_from': 38, 'up_thru': 85, 'down_at': 26, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6802', 'nonce': 52590}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6804', 'nonce': 52590}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6807', 'nonce': 52590}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6806', 'nonce': 52590}]}, 'public_addr': '172.21.15.138:6802/52590', 'cluster_addr': '172.21.15.138:6804/52590', 'heartbeat_back_addr': '172.21.15.138:6807/52590', 'heartbeat_front_addr': '172.21.15.138:6806/52590', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fd1fc124-7cf8-4628-9076-1f27d6385676', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 74, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6800', 'nonce': 43329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6801', 'nonce': 43329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6805', 'nonce': 43329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6803', 'nonce': 43329}]}, 'public_addr': '172.21.15.138:6800/43329', 'cluster_addr': '172.21.15.138:6801/43329', 'heartbeat_back_addr': '172.21.15.138:6805/43329', 'heartbeat_front_addr': '172.21.15.138:6803/43329', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '53ae8193-5c3e-4698-aacd-df2ad1db9143', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6812', 'nonce': 43351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6813', 'nonce': 43351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6815', 'nonce': 43351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6814', 'nonce': 43351}]}, 'public_addr': '172.21.15.138:6812/43351', 'cluster_addr': '172.21.15.138:6813/43351', 'heartbeat_back_addr': '172.21.15.138:6815/43351', 'heartbeat_front_addr': '172.21.15.138:6814/43351', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:51:25.390868+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:51:21.956838+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:51:37.987332+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:51:33.801631+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:51:25.375647+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:51:21.239839+0000', 'dead_epoch': 26}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:51:16.692530+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:51:20.543600+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.16', 'osds': [0, 3, 7]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 13, 'length': 1}, {'begin': 19, 'length': 1}, {'begin': 21, 'length': 2}, {'begin': 24, 'length': 1}, {'begin': 27, 'length': 2}, {'begin': 30, 'length': 1}]}], 'new_removed_snaps': [{'pool': 4, 'snaps': [{'begin': 27, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T02:51:43.700 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:51:43.701 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:1186: finishing set_chunk to oid 5 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:1186: got expected ENOENT (src dne) 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:1185: finishing set_chunk to oid 39 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 74 (ObjNum 182 snap 15 seq_num 182) dirty exists 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:1182: done (8 left) 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:1184: done (7 left) 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:1185: done (6 left) 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:1186: done (5 left) 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:1193: write oid 11 current snap is 19 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:1192: got (22) Invalid argument 2024-04-20T02:51:43.703 INFO:tasks.rados.rados.0.smithi105.stdout:1193: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-04-20T02:51:43.704 INFO:tasks.rados.rados.0.smithi105.stdout:1193: writing smithi10548823-11 from 729470 to 1273291 tid 1 2024-04-20T02:51:43.704 INFO:tasks.rados.rados.0.smithi105.stdout:1193: writing smithi10548823-11 from 1772395 to 2310463 tid 2 2024-04-20T02:51:43.704 INFO:tasks.rados.rados.0.smithi105.stdout:1193: writing smithi10548823-11 from 2900946 to 3303951 tid 3 2024-04-20T02:51:43.704 INFO:tasks.rados.rados.0.smithi105.stdout:1192: done (5 left) 2024-04-20T02:51:43.704 INFO:tasks.rados.rados.0.smithi105.stdout:1194: read oid 34 snap 17 2024-04-20T02:51:43.704 INFO:tasks.rados.rados.0.smithi105.stdout:1194: expect (ObjNum 170 snap 14 seq_num 170) 2024-04-20T02:51:43.704 INFO:tasks.rados.rados.0.smithi105.stdout:1195: snap_create 2024-04-20T02:51:43.704 INFO:tasks.rados.rados.0.smithi105.stdout:1188: expect (ObjNum 156 snap 10 seq_num 156) 2024-04-20T02:51:43.704 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:43.576+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:43.705 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:43.705 INFO:tasks.rados.rados.0.smithi105.stdout:1189: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:43.705 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:43.705 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:43.706 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:43.703+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:43.712 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:43.803 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:43.803+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:43.813 INFO:tasks.rados.rados.0.smithi105.stdout:1190: expect (ObjNum 146 snap 10 seq_num 146) 2024-04-20T02:51:43.839 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:43.839 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:43.839 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:43.839 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:43.849 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:43.904 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:43.903+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:43.932 INFO:tasks.rados.rados.0.smithi105.stdout:1191: finishing rollback tid 0 to smithi10548823-2 2024-04-20T02:51:43.932 INFO:tasks.rados.rados.0.smithi105.stdout:1191: finishing rollback tid 1 to smithi10548823-2 2024-04-20T02:51:43.932 INFO:tasks.rados.rados.0.smithi105.stdout:1191: finishing rollback tid 2 to smithi10548823-2 2024-04-20T02:51:43.932 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 142 (ObjNum 155 snap 10 seq_num 155) dirty exists 2024-04-20T02:51:43.933 INFO:tasks.rados.rados.0.smithi105.stdout:1194: expect (ObjNum 170 snap 14 seq_num 170) 2024-04-20T02:51:43.947 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:43.980 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:43.980 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:43.981 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:43.990 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:44.004 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:44.005+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:44.015 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:44.046 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:51:44.048 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:51:44.049 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:44.050 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:44.051 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:44.051 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:44.051 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:44.051 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:44.051 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:44.060 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-04-20T02:51:43.455682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":299711891,"num_objects":207,"num_object_clones":54,"num_object_copies":621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":4,"num_read":1789,"num_read_kb":1408209,"num_write":1317,"num_write_kb":356109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":97605550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":57,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":807916,"kb_used_data":566436,"kb_used_omap":12,"kb_used_meta":241331,"kb_avail":754166804,"statfs":{"total":773094113280,"available":772266807296,"internally_reserved":0,"allocated":580030464,"data_stored":657923725,"data_compressed":14011262,"data_compressed_allocated":14532608,"data_compressed_original":33570390,"omap_allocated":12807,"internal_metadata":247123449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27047828,"num_objects":-26,"num_object_clones":-9,"num_object_copies":-78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":2,"num_read":208,"num_read_kb":164964,"num_write":128,"num_write_kb":27334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":17174330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000784"},"pg_stats":[{"pgid":"4.17","version":"84'64","reported_seq":282,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:41.215689+0000","last_change":"2024-04-20T02:51:41.214474+0000","last_active":"2024-04-20T02:51:41.215689+0000","last_peered":"2024-04-20T02:51:41.215689+0000","last_clean":"2024-04-20T02:51:41.215689+0000","last_became_active":"2024-04-20T02:51:30.147180+0000","last_became_peered":"2024-04-20T02:51:30.147180+0000","last_unstale":"2024-04-20T02:51:41.215689+0000","last_undegraded":"2024-04-20T02:51:41.215689+0000","last_fullsized":"2024-04-20T02:51:41.215689+0000","mapping_epoch":72,"log_start":"80'57","ondisk_log_start":"80'57","created":60,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":54,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.077489709000000004,"stat_sum":{"num_bytes":9750895,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":55,"num_read_kb":73762,"num_write":32,"num_write_kb":9589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2757906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.1","version":"58'101","reported_seq":338,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-20T02:51:38.129998+0000","last_change":"2024-04-20T02:51:37.138356+0000","last_active":"2024-04-20T02:51:38.129998+0000","last_peered":"2024-04-20T02:51:38.129998+0000","last_clean":"2024-04-20T02:51:38.129998+0000","last_became_active":"2024-04-20T02:51:18.508043+0000","last_became_peered":"2024-04-20T02:51:18.508043+0000","last_unstale":"2024-04-20T02:51:38.129998+0000","last_undegraded":"2024-04-20T02:51:38.129998+0000","last_fullsized":"2024-04-20T02:51:38.129998+0000","mapping_epoch":46,"log_start":"57'97","ondisk_log_start":"57'97","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'101","last_scrub_stamp":"2024-04-20T02:51:34.098939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:34.098939+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":97,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:56.647060+0000","scrub_duration":0.0019988520000000002,"objects_trimmed":0,"snaptrim_duration":0.010431897000000001,"stat_sum":{"num_bytes":3479016,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":9926,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-20T02:51:41.123289+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:51:41.123289+0000","last_peered":"2024-04-20T02:51:41.123289+0000","last_clean":"2024-04-20T02:51:41.123289+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:51:41.123289+0000","last_undegraded":"2024-04-20T02:51:41.123289+0000","last_fullsized":"2024-04-20T02:51:41.123289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:45.432280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"85'81","reported_seq":309,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:42.124488+0000","last_change":"2024-04-20T02:51:41.126001+0000","last_active":"2024-04-20T02:51:42.124488+0000","last_peered":"2024-04-20T02:51:42.124488+0000","last_clean":"2024-04-20T02:51:42.124488+0000","last_became_active":"2024-04-20T02:51:18.505708+0000","last_became_peered":"2024-04-20T02:51:18.505708+0000","last_unstale":"2024-04-20T02:51:42.124488+0000","last_undegraded":"2024-04-20T02:51:42.124488+0000","last_fullsized":"2024-04-20T02:51:42.124488+0000","mapping_epoch":22,"log_start":"81'78","ondisk_log_start":"81'78","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":74,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.922550+0000","scrub_duration":0.001820667,"objects_trimmed":0,"snaptrim_duration":0.00100891,"stat_sum":{"num_bytes":9040558,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":17295,"num_write":8,"num_write_kb":2414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":76,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-20T02:51:41.125853+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:51:41.125853+0000","last_peered":"2024-04-20T02:51:41.125853+0000","last_clean":"2024-04-20T02:51:41.125853+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:51:41.125853+0000","last_undegraded":"2024-04-20T02:51:41.125853+0000","last_fullsized":"2024-04-20T02:51:41.125853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-20T02:51:38.131807+0000","last_change":"2024-04-20T02:50:28.091938+0000","last_active":"2024-04-20T02:51:38.131807+0000","last_peered":"2024-04-20T02:51:38.131807+0000","last_clean":"2024-04-20T02:51:38.131807+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:51:38.131807+0000","last_undegraded":"2024-04-20T02:51:38.131807+0000","last_fullsized":"2024-04-20T02:51:38.131807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"80'181","reported_seq":449,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-20T02:51:38.132539+0000","last_change":"2024-04-20T02:51:37.153885+0000","last_active":"2024-04-20T02:51:38.132539+0000","last_peered":"2024-04-20T02:51:38.132539+0000","last_clean":"2024-04-20T02:51:38.132539+0000","last_became_active":"2024-04-20T02:51:18.502004+0000","last_became_peered":"2024-04-20T02:51:18.502004+0000","last_unstale":"2024-04-20T02:51:38.132539+0000","last_undegraded":"2024-04-20T02:51:38.132539+0000","last_fullsized":"2024-04-20T02:51:38.132539+0000","mapping_epoch":47,"log_start":"79'177","ondisk_log_start":"79'177","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'157","last_scrub_stamp":"2024-04-20T02:51:19.377186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.377186+0000","objects_scrubbed":5,"log_size":4,"log_dups_size":173,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:38.408177+0000","scrub_duration":0.0026503899999999999,"objects_trimmed":0,"snaptrim_duration":0.020714992000000002,"stat_sum":{"num_bytes":10299190,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":31,"num_read_kb":52961,"num_write":10,"num_write_kb":2994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"3.1","version":"80'144","reported_seq":234,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-20T02:51:38.132171+0000","last_change":"2024-04-20T02:50:34.239216+0000","last_active":"2024-04-20T02:51:38.132171+0000","last_peered":"2024-04-20T02:51:38.132171+0000","last_clean":"2024-04-20T02:51:38.132171+0000","last_became_active":"2024-04-20T02:50:34.238906+0000","last_became_peered":"2024-04-20T02:50:34.238906+0000","last_unstale":"2024-04-20T02:51:38.132171+0000","last_undegraded":"2024-04-20T02:51:38.132171+0000","last_fullsized":"2024-04-20T02:51:38.132171+0000","mapping_epoch":21,"log_start":"78'141","ondisk_log_start":"78'141","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":166,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.390119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26128455,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":134,"num_read_kb":6415,"num_write":171,"num_write_kb":55257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":83,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-20T02:51:38.131783+0000","last_change":"2024-04-20T02:51:28.222310+0000","last_active":"2024-04-20T02:51:38.131783+0000","last_peered":"2024-04-20T02:51:38.131783+0000","last_clean":"2024-04-20T02:51:38.131783+0000","last_became_active":"2024-04-20T02:50:28.091570+0000","last_became_peered":"2024-04-20T02:50:28.091570+0000","last_unstale":"2024-04-20T02:51:38.131783+0000","last_undegraded":"2024-04-20T02:51:38.131783+0000","last_fullsized":"2024-04-20T02:51:38.131783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:28.222203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:28.222203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:39.650022+0000","scrub_duration":0.0013437259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"80'75","reported_seq":271,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-20T02:51:38.132361+0000","last_change":"2024-04-20T02:51:37.138338+0000","last_active":"2024-04-20T02:51:38.132361+0000","last_peered":"2024-04-20T02:51:38.132361+0000","last_clean":"2024-04-20T02:51:38.132361+0000","last_became_active":"2024-04-20T02:51:18.504889+0000","last_became_peered":"2024-04-20T02:51:18.504889+0000","last_unstale":"2024-04-20T02:51:38.132361+0000","last_undegraded":"2024-04-20T02:51:38.132361+0000","last_fullsized":"2024-04-20T02:51:38.132361+0000","mapping_epoch":22,"log_start":"78'71","ondisk_log_start":"78'71","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'57","last_scrub_stamp":"2024-04-20T02:51:21.386796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:21.386796+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":70,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:27.553181+0000","scrub_duration":0.0018878359999999999,"objects_trimmed":0,"snaptrim_duration":0.0077618339999999996,"stat_sum":{"num_bytes":5502855,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":11429,"num_write":6,"num_write_kb":1127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"3.2","version":"85'130","reported_seq":306,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-20T02:51:42.179827+0000","last_change":"2024-04-20T02:51:07.118974+0000","last_active":"2024-04-20T02:51:42.179827+0000","last_peered":"2024-04-20T02:51:42.179827+0000","last_clean":"2024-04-20T02:51:42.179827+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:51:42.179827+0000","last_undegraded":"2024-04-20T02:51:42.179827+0000","last_fullsized":"2024-04-20T02:51:42.179827+0000","mapping_epoch":49,"log_start":"85'127","ondisk_log_start":"85'127","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":135,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:42.672805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24162553,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":117,"num_read_kb":8276,"num_write":147,"num_write_kb":57375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":225,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-20T02:51:38.131810+0000","last_change":"2024-04-20T02:50:19.923431+0000","last_active":"2024-04-20T02:51:38.131810+0000","last_peered":"2024-04-20T02:51:38.131810+0000","last_clean":"2024-04-20T02:51:38.131810+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:51:38.131810+0000","last_undegraded":"2024-04-20T02:51:38.131810+0000","last_fullsized":"2024-04-20T02:51:38.131810+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:50:17.569030+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-20T02:51:42.124138+0000","last_change":"2024-04-20T02:51:30.923927+0000","last_active":"2024-04-20T02:51:42.124138+0000","last_peered":"2024-04-20T02:51:42.124138+0000","last_clean":"2024-04-20T02:51:42.124138+0000","last_became_active":"2024-04-20T02:51:01.151544+0000","last_became_peered":"2024-04-20T02:51:01.151544+0000","last_unstale":"2024-04-20T02:51:42.124138+0000","last_undegraded":"2024-04-20T02:51:42.124138+0000","last_fullsized":"2024-04-20T02:51:42.124138+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:30.923833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:30.923833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:47.551592+0000","scrub_duration":0.001311373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"56'33","reported_seq":217,"reported_epoch":84,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-20T02:51:41.123324+0000","last_change":"2024-04-20T02:51:41.123301+0000","last_active":"2024-04-20T02:51:41.123324+0000","last_peered":"2024-04-20T02:51:41.123324+0000","last_clean":"2024-04-20T02:51:41.123324+0000","last_became_active":"2024-04-20T02:51:18.502179+0000","last_became_peered":"2024-04-20T02:51:18.502179+0000","last_unstale":"2024-04-20T02:51:41.123324+0000","last_undegraded":"2024-04-20T02:51:41.123324+0000","last_fullsized":"2024-04-20T02:51:41.123324+0000","mapping_epoch":22,"log_start":"53'30","ondisk_log_start":"53'30","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'33","last_scrub_stamp":"2024-04-20T02:51:32.144366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.144366+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:49.222704+0000","scrub_duration":0.01361505,"objects_trimmed":0,"snaptrim_duration":0.00051368200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"3.0","version":"85'120","reported_seq":201,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-20T02:51:42.177123+0000","last_change":"2024-04-20T02:50:34.241909+0000","last_active":"2024-04-20T02:51:42.177123+0000","last_peered":"2024-04-20T02:51:42.177123+0000","last_clean":"2024-04-20T02:51:42.177123+0000","last_became_active":"2024-04-20T02:50:34.241613+0000","last_became_peered":"2024-04-20T02:50:34.241613+0000","last_unstale":"2024-04-20T02:51:42.177123+0000","last_undegraded":"2024-04-20T02:51:42.177123+0000","last_fullsized":"2024-04-20T02:51:42.177123+0000","mapping_epoch":21,"log_start":"85'117","ondisk_log_start":"85'117","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":129,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:54.710965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20478244,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":104,"num_read_kb":3308,"num_write":149,"num_write_kb":44170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":85,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-20T02:51:41.175139+0000","last_change":"2024-04-20T02:51:41.175139+0000","last_active":"2024-04-20T02:51:41.175139+0000","last_peered":"2024-04-20T02:51:41.175139+0000","last_clean":"2024-04-20T02:51:41.175139+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:51:41.175139+0000","last_undegraded":"2024-04-20T02:51:41.175139+0000","last_fullsized":"2024-04-20T02:51:41.175139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:41.175035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:41.175035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:46.711136+0000","scrub_duration":0.0014447850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'56","reported_seq":270,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:41.213116+0000","last_change":"2024-04-20T02:51:41.211721+0000","last_active":"2024-04-20T02:51:41.213116+0000","last_peered":"2024-04-20T02:51:41.213116+0000","last_clean":"2024-04-20T02:51:41.213116+0000","last_became_active":"2024-04-20T02:51:18.506025+0000","last_became_peered":"2024-04-20T02:51:18.506025+0000","last_unstale":"2024-04-20T02:51:41.213116+0000","last_undegraded":"2024-04-20T02:51:41.213116+0000","last_fullsized":"2024-04-20T02:51:41.213116+0000","mapping_epoch":46,"log_start":"76'52","ondisk_log_start":"76'52","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'50","last_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.487770+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":52,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:38.165123+0000","scrub_duration":0.019136573,"objects_trimmed":0,"snaptrim_duration":0.077158452000000002,"stat_sum":{"num_bytes":6405010,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"3.3","version":"82'111","reported_seq":194,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-20T02:51:39.175745+0000","last_change":"2024-04-20T02:50:34.242061+0000","last_active":"2024-04-20T02:51:39.175745+0000","last_peered":"2024-04-20T02:51:39.175745+0000","last_clean":"2024-04-20T02:51:39.175745+0000","last_became_active":"2024-04-20T02:50:34.241638+0000","last_became_peered":"2024-04-20T02:50:34.241638+0000","last_unstale":"2024-04-20T02:51:39.175745+0000","last_undegraded":"2024-04-20T02:51:39.175745+0000","last_fullsized":"2024-04-20T02:51:39.175745+0000","mapping_epoch":21,"log_start":"78'108","ondisk_log_start":"78'108","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:58.050023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30340282,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":100,"num_read_kb":4900,"num_write":136,"num_write_kb":43111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-20T02:51:42.124099+0000","last_change":"2024-04-20T02:51:01.134074+0000","last_active":"2024-04-20T02:51:42.124099+0000","last_peered":"2024-04-20T02:51:42.124099+0000","last_clean":"2024-04-20T02:51:42.124099+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:51:42.124099+0000","last_undegraded":"2024-04-20T02:51:42.124099+0000","last_fullsized":"2024-04-20T02:51:42.124099+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"56'17","reported_seq":183,"reported_epoch":84,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-20T02:51:41.126140+0000","last_change":"2024-04-20T02:51:39.207504+0000","last_active":"2024-04-20T02:51:41.126140+0000","last_peered":"2024-04-20T02:51:41.126140+0000","last_clean":"2024-04-20T02:51:41.126140+0000","last_became_active":"2024-04-20T02:51:18.509276+0000","last_became_peered":"2024-04-20T02:51:18.509276+0000","last_unstale":"2024-04-20T02:51:41.126140+0000","last_undegraded":"2024-04-20T02:51:41.126140+0000","last_fullsized":"2024-04-20T02:51:41.126140+0000","mapping_epoch":22,"log_start":"53'14","ondisk_log_start":"53'14","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:51:22.420320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:22.420320+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.257627+0000","scrub_duration":0.0016858400000000001,"objects_trimmed":0,"snaptrim_duration":0.077899140000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.2","version":"0'0","reported_seq":168,"reported_epoch":84,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-20T02:51:41.211634+0000","last_change":"2024-04-20T02:51:41.211634+0000","last_active":"2024-04-20T02:51:41.211634+0000","last_peered":"2024-04-20T02:51:41.211634+0000","last_clean":"2024-04-20T02:51:41.211634+0000","last_became_active":"2024-04-20T02:51:18.506462+0000","last_became_peered":"2024-04-20T02:51:18.506462+0000","last_unstale":"2024-04-20T02:51:41.211634+0000","last_undegraded":"2024-04-20T02:51:41.211634+0000","last_fullsized":"2024-04-20T02:51:41.211634+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:33.440257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:51:33.440257+0000","last_clean_scrub_stamp":"2024-04-20T02:51:33.440257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:00.147584+0000","scrub_duration":0.001747176,"objects_trimmed":0,"snaptrim_duration":0.085723031000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":76,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-04-20T02:51:41.126039+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:51:41.126039+0000","last_peered":"2024-04-20T02:51:41.126039+0000","last_clean":"2024-04-20T02:51:41.126039+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:51:41.126039+0000","last_undegraded":"2024-04-20T02:51:41.126039+0000","last_fullsized":"2024-04-20T02:51:41.126039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"86'65","reported_seq":265,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-20T02:51:43.145406+0000","last_change":"2024-04-20T02:51:43.145610+0000","last_active":"2024-04-20T02:51:43.145406+0000","last_peered":"2024-04-20T02:51:43.145406+0000","last_clean":"2024-04-20T02:51:43.145406+0000","last_became_active":"2024-04-20T02:51:43.136495+0000","last_became_peered":"2024-04-20T02:51:43.136495+0000","last_unstale":"2024-04-20T02:51:43.145406+0000","last_undegraded":"2024-04-20T02:51:43.145406+0000","last_fullsized":"2024-04-20T02:51:43.145406+0000","mapping_epoch":62,"log_start":"82'63","ondisk_log_start":"82'63","created":22,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'47","last_scrub_stamp":"2024-04-20T02:51:28.917651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:28.917651+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":107,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001846091,"objects_trimmed":0,"snaptrim_duration":0.0084690749999999995,"stat_sum":{"num_bytes":4692590,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":110606,"num_write":25,"num_write_kb":6120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8110636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"2"},{"start":"1e","length":"1"}]},{"pgid":"4.9","version":"82'196","reported_seq":441,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-20T02:51:39.150449+0000","last_change":"2024-04-20T02:51:39.150449+0000","last_active":"2024-04-20T02:51:39.150449+0000","last_peered":"2024-04-20T02:51:39.150449+0000","last_clean":"2024-04-20T02:51:39.150449+0000","last_became_active":"2024-04-20T02:51:39.132079+0000","last_became_peered":"2024-04-20T02:51:39.132079+0000","last_unstale":"2024-04-20T02:51:39.150449+0000","last_undegraded":"2024-04-20T02:51:39.150449+0000","last_fullsized":"2024-04-20T02:51:39.150449+0000","mapping_epoch":22,"log_start":"82'193","ondisk_log_start":"82'193","created":22,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'173","last_scrub_stamp":"2024-04-20T02:51:33.090712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:33.090712+0000","objects_scrubbed":10,"log_size":3,"log_dups_size":361,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051817219999999997,"objects_trimmed":0,"snaptrim_duration":0.017698746000000001,"stat_sum":{"num_bytes":29236168,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":125,"num_read_kb":125731,"num_write":60,"num_write_kb":15897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":17174330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.a","version":"84'50","reported_seq":229,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:41.125187+0000","last_change":"2024-04-20T02:51:41.123393+0000","last_active":"2024-04-20T02:51:41.125187+0000","last_peered":"2024-04-20T02:51:41.125187+0000","last_clean":"2024-04-20T02:51:41.125187+0000","last_became_active":"2024-04-20T02:50:35.413503+0000","last_became_peered":"2024-04-20T02:50:35.413503+0000","last_unstale":"2024-04-20T02:51:41.125187+0000","last_undegraded":"2024-04-20T02:51:41.125187+0000","last_fullsized":"2024-04-20T02:51:41.125187+0000","mapping_epoch":22,"log_start":"78'46","ondisk_log_start":"78'46","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":46,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:57.185462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039962500000000002,"stat_sum":{"num_bytes":6858517,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":126157,"num_write":32,"num_write_kb":8571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"4.b","version":"82'123","reported_seq":304,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-20T02:51:39.176252+0000","last_change":"2024-04-20T02:51:37.148126+0000","last_active":"2024-04-20T02:51:39.176252+0000","last_peered":"2024-04-20T02:51:39.176252+0000","last_clean":"2024-04-20T02:51:39.176252+0000","last_became_active":"2024-04-20T02:50:35.242331+0000","last_became_peered":"2024-04-20T02:50:35.242331+0000","last_unstale":"2024-04-20T02:51:39.176252+0000","last_undegraded":"2024-04-20T02:51:39.176252+0000","last_fullsized":"2024-04-20T02:51:39.176252+0000","mapping_epoch":22,"log_start":"80'120","ondisk_log_start":"80'120","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":120,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.773051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015130737999999999,"stat_sum":{"num_bytes":17909477,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":146,"num_read_kb":175674,"num_write":81,"num_write_kb":21776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.c","version":"80'67","reported_seq":213,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-20T02:51:38.129765+0000","last_change":"2024-04-20T02:51:37.130942+0000","last_active":"2024-04-20T02:51:38.129765+0000","last_peered":"2024-04-20T02:51:38.129765+0000","last_clean":"2024-04-20T02:51:38.129765+0000","last_became_active":"2024-04-20T02:50:35.241480+0000","last_became_peered":"2024-04-20T02:50:35.241480+0000","last_unstale":"2024-04-20T02:51:38.129765+0000","last_undegraded":"2024-04-20T02:51:38.129765+0000","last_fullsized":"2024-04-20T02:51:38.129765+0000","mapping_epoch":22,"log_start":"79'63","ondisk_log_start":"79'63","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":63,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.915698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036758559999999999,"stat_sum":{"num_bytes":8559939,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":50,"num_read_kb":49915,"num_write":41,"num_write_kb":11612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.d","version":"82'127","reported_seq":326,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-20T02:51:39.276539+0000","last_change":"2024-04-20T02:51:37.136664+0000","last_active":"2024-04-20T02:51:39.276539+0000","last_peered":"2024-04-20T02:51:39.276539+0000","last_clean":"2024-04-20T02:51:39.276539+0000","last_became_active":"2024-04-20T02:50:35.528107+0000","last_became_peered":"2024-04-20T02:50:35.528107+0000","last_unstale":"2024-04-20T02:51:39.276539+0000","last_undegraded":"2024-04-20T02:51:39.276539+0000","last_fullsized":"2024-04-20T02:51:39.276539+0000","mapping_epoch":22,"log_start":"80'123","ondisk_log_start":"80'123","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":123,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:59.747989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091383060000000006,"stat_sum":{"num_bytes":17655284,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":217,"num_read_kb":243189,"num_write":77,"num_write_kb":21550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.e","version":"80'58","reported_seq":203,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-20T02:51:38.129647+0000","last_change":"2024-04-20T02:51:37.136764+0000","last_active":"2024-04-20T02:51:38.129647+0000","last_peered":"2024-04-20T02:51:38.129647+0000","last_clean":"2024-04-20T02:51:38.129647+0000","last_became_active":"2024-04-20T02:50:35.244872+0000","last_became_peered":"2024-04-20T02:50:35.244872+0000","last_unstale":"2024-04-20T02:51:38.129647+0000","last_undegraded":"2024-04-20T02:51:38.129647+0000","last_fullsized":"2024-04-20T02:51:38.129647+0000","mapping_epoch":22,"log_start":"78'54","ondisk_log_start":"78'54","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":54,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:39.148211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094314689999999996,"stat_sum":{"num_bytes":9356935,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":36172,"num_write":34,"num_write_kb":11852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.f","version":"82'22","reported_seq":161,"reported_epoch":84,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-20T02:51:41.214480+0000","last_change":"2024-04-20T02:51:41.214480+0000","last_active":"2024-04-20T02:51:41.214480+0000","last_peered":"2024-04-20T02:51:41.214480+0000","last_clean":"2024-04-20T02:51:41.214480+0000","last_became_active":"2024-04-20T02:50:35.245323+0000","last_became_peered":"2024-04-20T02:50:35.245323+0000","last_unstale":"2024-04-20T02:51:41.214480+0000","last_undegraded":"2024-04-20T02:51:41.214480+0000","last_fullsized":"2024-04-20T02:51:41.214480+0000","mapping_epoch":22,"log_start":"34'15","ondisk_log_start":"34'15","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":15,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:53.742736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.087844001000000005,"stat_sum":{"num_bytes":4849918,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":19399,"num_write":19,"num_write_kb":5420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.10","version":"84'89","reported_seq":345,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:41.152670+0000","last_change":"2024-04-20T02:51:41.125815+0000","last_active":"2024-04-20T02:51:41.152670+0000","last_peered":"2024-04-20T02:51:41.152670+0000","last_clean":"2024-04-20T02:51:41.152670+0000","last_became_active":"2024-04-20T02:51:30.132114+0000","last_became_peered":"2024-04-20T02:51:30.132114+0000","last_unstale":"2024-04-20T02:51:41.152670+0000","last_undegraded":"2024-04-20T02:51:41.152670+0000","last_fullsized":"2024-04-20T02:51:41.152670+0000","mapping_epoch":72,"log_start":"83'86","ondisk_log_start":"83'86","created":60,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'78","last_scrub_stamp":"2024-04-20T02:51:32.278907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.278907+0000","objects_scrubbed":5,"log_size":3,"log_dups_size":84,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:41.876076+0000","scrub_duration":0.0025619089999999998,"objects_trimmed":0,"snaptrim_duration":0.00057729999999999999,"stat_sum":{"num_bytes":9481621,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":28163,"num_write":2,"num_write_kb":466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"4.11","version":"80'135","reported_seq":442,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-20T02:51:39.217593+0000","last_change":"2024-04-20T02:51:37.145300+0000","last_active":"2024-04-20T02:51:39.217593+0000","last_peered":"2024-04-20T02:51:39.217593+0000","last_clean":"2024-04-20T02:51:39.217593+0000","last_became_active":"2024-04-20T02:51:32.123982+0000","last_became_peered":"2024-04-20T02:51:32.123982+0000","last_unstale":"2024-04-20T02:51:39.217593+0000","last_undegraded":"2024-04-20T02:51:39.217593+0000","last_fullsized":"2024-04-20T02:51:39.217593+0000","mapping_epoch":74,"log_start":"79'131","ondisk_log_start":"79'131","created":60,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":131,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017519818999999999,"stat_sum":{"num_bytes":9642040,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":73,"num_read_kb":75893,"num_write":57,"num_write_kb":15124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":30043684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.12","version":"0'0","reported_seq":174,"reported_epoch":84,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-20T02:51:41.125659+0000","last_change":"2024-04-20T02:51:39.208011+0000","last_active":"2024-04-20T02:51:41.125659+0000","last_peered":"2024-04-20T02:51:41.125659+0000","last_clean":"2024-04-20T02:51:41.125659+0000","last_became_active":"2024-04-20T02:51:29.138939+0000","last_became_peered":"2024-04-20T02:51:29.138939+0000","last_unstale":"2024-04-20T02:51:41.125659+0000","last_undegraded":"2024-04-20T02:51:41.125659+0000","last_fullsized":"2024-04-20T02:51:41.125659+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.078770568999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.13","version":"80'192","reported_seq":492,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-20T02:51:38.132217+0000","last_change":"2024-04-20T02:51:37.145554+0000","last_active":"2024-04-20T02:51:38.132217+0000","last_peered":"2024-04-20T02:51:38.132217+0000","last_clean":"2024-04-20T02:51:38.132217+0000","last_became_active":"2024-04-20T02:51:29.136510+0000","last_became_peered":"2024-04-20T02:51:29.136510+0000","last_unstale":"2024-04-20T02:51:38.132217+0000","last_undegraded":"2024-04-20T02:51:38.132217+0000","last_fullsized":"2024-04-20T02:51:38.132217+0000","mapping_epoch":71,"log_start":"79'186","ondisk_log_start":"79'186","created":60,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'167","last_scrub_stamp":"2024-04-20T02:51:33.169112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:33.169112+0000","objects_scrubbed":10,"log_size":6,"log_dups_size":186,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:39.831181+0000","scrub_duration":0.015460714,"objects_trimmed":0,"snaptrim_duration":0.01270164,"stat_sum":{"num_bytes":16829955,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":26,"num_read_kb":31690,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.14","version":"82'26","reported_seq":166,"reported_epoch":84,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-20T02:51:41.211277+0000","last_change":"2024-04-20T02:51:41.211277+0000","last_active":"2024-04-20T02:51:41.211277+0000","last_peered":"2024-04-20T02:51:41.211277+0000","last_clean":"2024-04-20T02:51:41.211277+0000","last_became_active":"2024-04-20T02:51:30.144817+0000","last_became_peered":"2024-04-20T02:51:30.144817+0000","last_unstale":"2024-04-20T02:51:41.211277+0000","last_undegraded":"2024-04-20T02:51:41.211277+0000","last_fullsized":"2024-04-20T02:51:41.211277+0000","mapping_epoch":72,"log_start":"64'22","ondisk_log_start":"64'22","created":60,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":22,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.085249380999999999,"stat_sum":{"num_bytes":3093185,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":26722,"num_write":10,"num_write_kb":2705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"4.15","version":"84'52","reported_seq":261,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:41.166704+0000","last_change":"2024-04-20T02:51:41.123461+0000","last_active":"2024-04-20T02:51:41.166704+0000","last_peered":"2024-04-20T02:51:41.166704+0000","last_clean":"2024-04-20T02:51:41.166704+0000","last_became_active":"2024-04-20T02:51:32.124965+0000","last_became_peered":"2024-04-20T02:51:32.124965+0000","last_unstale":"2024-04-20T02:51:41.166704+0000","last_undegraded":"2024-04-20T02:51:41.166704+0000","last_fullsized":"2024-04-20T02:51:41.166704+0000","mapping_epoch":74,"log_start":"80'49","ondisk_log_start":"80'49","created":60,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000452886,"stat_sum":{"num_bytes":5622126,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":92387,"num_write":22,"num_write_kb":6348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":16438198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"4.16","version":"73'67","reported_seq":294,"reported_epoch":81,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:38.132216+0000","last_change":"2024-04-20T02:51:30.147512+0000","last_active":"2024-04-20T02:51:38.132216+0000","last_peered":"2024-04-20T02:51:38.132216+0000","last_clean":"2024-04-20T02:51:27.811685+0000","last_became_active":"2024-04-20T02:51:30.142488+0000","last_became_peered":"2024-04-20T02:51:30.142488+0000","last_unstale":"2024-04-20T02:51:38.132216+0000","last_undegraded":"2024-04-20T02:51:38.132216+0000","last_fullsized":"2024-04-20T02:51:38.132216+0000","mapping_epoch":72,"log_start":"73'64","ondisk_log_start":"73'64","created":60,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099853799999999999,"stat_sum":{"num_bytes":9746710,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":73,"num_read_kb":78139,"num_write":38,"num_write_kb":10215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8853601,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":24,"stat_sum":{"num_bytes":198011989,"num_objects":95,"num_object_clones":54,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":4,"num_read":1218,"num_read_kb":1385210,"num_write":555,"num_write_kb":153780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":83378355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":57,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":391057408,"data_stored":453785468,"data_compressed":1878984,"data_compressed_allocated":1454080,"data_compressed_original":4103766,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":101109534,"num_objects":110,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":455,"num_read_kb":22899,"num_write":603,"num_write_kb":199913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163201024,"data_stored":177028223,"data_compressed":11501305,"data_compressed_allocated":12324864,"data_compressed_original":26619904,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509462,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86316,"kb_used_data":55192,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94285524,"statfs":{"total":96636764160,"available":96548376576,"internally_reserved":0,"allocated":56516608,"data_stored":63435971,"data_compressed":295417,"data_compressed_allocated":294912,"data_compressed_original":1064960,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.33200000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.44800000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.35699999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.435},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.42199999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.29799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.52900000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757259,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78932,"kb_used_data":51200,"kb_used_omap":1,"kb_used_meta":27710,"kb_avail":94292908,"statfs":{"total":96636764160,"available":96555937792,"internally_reserved":0,"allocated":52428800,"data_stored":58294149,"data_compressed":988712,"data_compressed_allocated":1060864,"data_compressed_original":2211840,"omap_allocated":1657,"internal_metadata":28375431},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61264,"kb_used_data":34244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94310576,"statfs":{"total":96636764160,"available":96574029824,"internally_reserved":0,"allocated":35065856,"data_stored":40369088,"data_compressed":1156356,"data_compressed_allocated":1179648,"data_compressed_original":2555904,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.23000000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.218},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.26400000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.28299999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":11.877000000000001,"5min":11.877000000000001,"15min":11.877000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.34699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":11.805,"5min":11.805,"15min":11.805},"last":0.41799999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.34000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101084,"kb_used_data":69968,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94270756,"statfs":{"total":96636764160,"available":96533254144,"internally_reserved":0,"allocated":71647232,"data_stored":83382927,"data_compressed":1168975,"data_compressed_allocated":1167360,"data_compressed_original":2686976,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.36499999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.43099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.52400000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54500000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100608,"kb_used_data":69488,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94271232,"statfs":{"total":96636764160,"available":96533741568,"internally_reserved":0,"allocated":71155712,"data_stored":78112383,"data_compressed":2163477,"data_compressed_allocated":2265088,"data_compressed_original":4775936,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.371}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.313}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.23799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.504},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.51100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.42699999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.53300000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118872,"kb_used_data":87748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94252968,"statfs":{"total":96636764160,"available":96515039232,"internally_reserved":0,"allocated":89853952,"data_stored":98408973,"data_compressed":3472251,"data_compressed_allocated":3747840,"data_compressed_original":8466432,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.58599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.495},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.0369999999999999,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.432}]},{"osd":3,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":9.9570000000000007,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.47899999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.8679999999999999,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.98,"5min":3.98,"15min":3.98},"last":0.60499999999999998}]}]},{"osd":0,"up_from":15,"seq":64424509458,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123628,"kb_used_data":92504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94248212,"statfs":{"total":96636764160,"available":96510169088,"internally_reserved":0,"allocated":94724096,"data_stored":111009106,"data_compressed":2423688,"data_compressed_allocated":2441216,"data_compressed_original":6234112,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.626},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.216}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.59299999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.56399999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.372}]}]},{"osd":4,"up_from":15,"seq":64424509460,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137212,"kb_used_data":106092,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94234628,"statfs":{"total":96636764160,"available":96496259072,"internally_reserved":0,"allocated":108638208,"data_stored":124911128,"data_compressed":2342386,"data_compressed_allocated":2375680,"data_compressed_original":5574230,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.25900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.501},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.53600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.40000000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.41799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31510528,"data_stored":34124552,"data_compressed":2084339,"data_compressed_allocated":2236416,"data_compressed_original":4935680,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11526144,"data_stored":12519397,"data_compressed":873572,"data_compressed_allocated":933888,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11526144,"data_stored":12519397,"data_compressed":873572,"data_compressed_allocated":933888,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26750976,"data_stored":28901823,"data_compressed":1873378,"data_compressed_allocated":2002944,"data_compressed_original":4222976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":31510528,"data_stored":34124552,"data_compressed":2084339,"data_compressed_allocated":2236416,"data_compressed_original":4935680,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11968512,"data_stored":12993166,"data_compressed":953944,"data_compressed_allocated":1019904,"data_compressed_original":2080768,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":38408192,"data_stored":41845336,"data_compressed":2758161,"data_compressed_allocated":2961408,"data_compressed_original":6512640,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":63053824,"data_stored":76878262,"data_compressed":422212,"data_compressed_allocated":266240,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59150336,"data_stored":70446331,"data_compressed":295403,"data_compressed_allocated":233472,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15691776,"data_stored":18841772,"data_compressed":108740,"data_compressed_allocated":57344,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":43433984,"data_stored":48793361,"data_compressed":290099,"data_compressed_allocated":262144,"data_compressed_original":552960,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77590528,"data_stored":91431814,"data_compressed":346228,"data_compressed_allocated":233472,"data_compressed_original":835158,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":39489536,"data_stored":44883784,"data_compressed":34768,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":37105664,"data_stored":39997969,"data_compressed":91435,"data_compressed_allocated":98304,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":55541760,"data_stored":62512175,"data_compressed":290099,"data_compressed_allocated":262144,"data_compressed_original":552960,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:51:44.063 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [6, 0, 2] 2024-04-20T02:51:44.063 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.0', '6', '0', '2'] 2024-04-20T02:51:44.063 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.0 6 0 2 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:1193: finishing write tid 1 to smithi10548823-11 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:1193: finishing write tid 2 to smithi10548823-11 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:1193: finishing write tid 3 to smithi10548823-11 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:1193: finishing write tid 4 to smithi10548823-11 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:1193: finishing write tid 5 to smithi10548823-11 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:1193: oid 11 updating version 0 to 87 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:1193: oid 11 updating version 87 to 88 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:1193: oid 11 version 88 is already newer than 86 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:1193: oid 11 version 88 is already newer than 85 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 88 (ObjNum 210 snap 19 seq_num 210) dirty exists 2024-04-20T02:51:44.070 INFO:tasks.rados.rados.0.smithi105.stdout:1193: left oid 11 (ObjNum 210 snap 19 seq_num 210) 2024-04-20T02:51:44.113 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:44.113+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:44.161 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:44.161 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:44.171 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:44.206 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:44.207+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:44.214 INFO:tasks.rados.rados.0.smithi105.stdout:1188: done (6 left) 2024-04-20T02:51:44.214 INFO:tasks.rados.rados.0.smithi105.stdout:1189: done (5 left) 2024-04-20T02:51:44.214 INFO:tasks.rados.rados.0.smithi105.stdout:1190: done (4 left) 2024-04-20T02:51:44.214 INFO:tasks.rados.rados.0.smithi105.stdout:1191: done (3 left) 2024-04-20T02:51:44.214 INFO:tasks.rados.rados.0.smithi105.stdout:1193: done (2 left) 2024-04-20T02:51:44.214 INFO:tasks.rados.rados.0.smithi105.stdout:1194: done (1 left) 2024-04-20T02:51:44.214 INFO:tasks.rados.rados.0.smithi105.stdout:1195: done (0 left) 2024-04-20T02:51:44.214 INFO:tasks.rados.rados.0.smithi105.stdout:1196: rollback oid 50 current snap is 20 2024-04-20T02:51:44.215 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 50 to 19 2024-04-20T02:51:44.215 INFO:tasks.rados.rados.0.smithi105.stdout:1197: write (excl) oid 3 current snap is 20 2024-04-20T02:51:44.215 INFO:tasks.rados.rados.0.smithi105.stdout:1197: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-04-20T02:51:44.221 INFO:tasks.rados.rados.0.smithi105.stdout:1197: writing smithi10548823-3 from 746277 to 1365347 tid 1 2024-04-20T02:51:44.228 INFO:tasks.rados.rados.0.smithi105.stdout:1197: writing smithi10548823-3 from 1830877 to 2512603 tid 2 2024-04-20T02:51:44.234 INFO:tasks.rados.rados.0.smithi105.stdout:1197: writing smithi10548823-3 from 2928369 to 3546277 tid 3 2024-04-20T02:51:44.234 INFO:tasks.rados.rados.0.smithi105.stdout:1198: snap_remove snap 17 2024-04-20T02:51:44.308 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:44.307+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:44.308 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:44.308 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:44.318 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:44.408 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:44.408+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:44.488 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:44.488 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:44.498 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:44.509 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:44.510+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:44.610 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:44.611+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:44.628 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:44.628 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:44.639 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:44.711 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:44.711+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:44.768 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:44.769 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:44.779 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:44.812 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:44.812+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:44.913 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:44.913+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:44.948 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:44.948 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:44.959 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:45.014 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:45.015+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:45.115 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:45.116+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:45.129 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:45.129 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:45.140 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:45.160 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:51:45.215 INFO:tasks.rados.rados.0.smithi105.stdout:1198: done (2 left) 2024-04-20T02:51:45.215 INFO:tasks.rados.rados.0.smithi105.stdout:1199: tier_evict oid 47 2024-04-20T02:51:45.215 INFO:tasks.rados.rados.0.smithi105.stdout:1199: tier_evict oid 47 snap -1 2024-04-20T02:51:45.215 INFO:tasks.rados.rados.0.smithi105.stdout:1196: finishing rollback tid 0 to smithi10548823-50 2024-04-20T02:51:45.215 INFO:tasks.rados.rados.0.smithi105.stdout:1196: finishing rollback tid 1 to smithi10548823-50 2024-04-20T02:51:45.215 INFO:tasks.rados.rados.0.smithi105.stdout:1196: finishing rollback tid 2 to smithi10548823-50 2024-04-20T02:51:45.215 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 72 (ObjNum 146 snap 10 seq_num 146) dirty exists 2024-04-20T02:51:45.215 INFO:tasks.rados.rados.0.smithi105.stdout:1197: finishing write tid 1 to smithi10548823-3 2024-04-20T02:51:45.215 INFO:tasks.rados.rados.0.smithi105.stdout:1196: done (2 left) 2024-04-20T02:51:45.215 INFO:tasks.rados.rados.0.smithi105.stdout:1200: delete oid 18 current snap is 20 2024-04-20T02:51:45.216 INFO:tasks.rados.rados.0.smithi105.stdout:1197: finishing write tid 2 to smithi10548823-3 2024-04-20T02:51:45.216 INFO:tasks.rados.rados.0.smithi105.stdout:1197: finishing write tid 3 to smithi10548823-3 2024-04-20T02:51:45.216 INFO:tasks.rados.rados.0.smithi105.stdout:1197: finishing write tid 4 to smithi10548823-3 2024-04-20T02:51:45.216 INFO:tasks.rados.rados.0.smithi105.stdout:1197: finishing write tid 5 to smithi10548823-3 2024-04-20T02:51:45.216 INFO:tasks.rados.rados.0.smithi105.stdout:1197: oid 3 updating version 0 to 235 2024-04-20T02:51:45.216 INFO:tasks.rados.rados.0.smithi105.stdout:1197: oid 3 updating version 235 to 238 2024-04-20T02:51:45.216 INFO:tasks.rados.rados.0.smithi105.stdout:1197: oid 3 version 238 is already newer than 236 2024-04-20T02:51:45.216 INFO:tasks.rados.rados.0.smithi105.stdout:1197: oid 3 version 238 is already newer than 237 2024-04-20T02:51:45.216 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 238 (ObjNum 211 snap 20 seq_num 211) dirty exists 2024-04-20T02:51:45.216 INFO:tasks.rados.rados.0.smithi105.stdout:1197: left oid 3 (ObjNum 211 snap 20 seq_num 211) 2024-04-20T02:51:45.217 INFO:tasks.rados.rados.0.smithi105.stdout:1199: got (22) Invalid argument 2024-04-20T02:51:45.220 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:45.220+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:45.233 INFO:tasks.rados.rados.0.smithi105.stdout:1197: done (2 left) 2024-04-20T02:51:45.233 INFO:tasks.rados.rados.0.smithi105.stdout:1199: done (1 left) 2024-04-20T02:51:45.233 INFO:tasks.rados.rados.0.smithi105.stdout:1200: done (0 left) 2024-04-20T02:51:45.233 INFO:tasks.rados.rados.0.smithi105.stdout:1201: write oid 43 current snap is 20 2024-04-20T02:51:45.233 INFO:tasks.rados.rados.0.smithi105.stdout:1201: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-04-20T02:51:45.233 INFO:tasks.rados.rados.0.smithi105.stdout:1201: writing smithi10548823-43 from 779891 to 1265812 tid 1 2024-04-20T02:51:45.242 INFO:tasks.rados.rados.0.smithi105.stdout:1201: writing smithi10548823-43 from 1947841 to 2633236 tid 2 2024-04-20T02:51:45.245 INFO:teuthology.orchestra.run.smithi105.stderr:set 2.0 pg_upmap mapping to [6,0,2] 2024-04-20T02:51:45.248 INFO:tasks.rados.rados.0.smithi105.stdout:1201: writing smithi10548823-43 from 3383215 to 3579891 tid 3 2024-04-20T02:51:45.248 INFO:tasks.rados.rados.0.smithi105.stdout:1202: set_chunk oid 35 target oid 2024-04-20T02:51:45.285 INFO:tasks.rados.rados.0.smithi105.stdout:1202: set_chunk oid 35 offset: 744448 length: 1264640 target oid 863bbbb0c9c99a18721ce4fa6300a44c6824c098478cb7cbb98005bc91f553ba offset: 0 snap -1 2024-04-20T02:51:45.285 INFO:tasks.rados.rados.0.smithi105.stdout:1203: tier_evict oid 45 2024-04-20T02:51:45.285 INFO:tasks.rados.rados.0.smithi105.stdout:1203: tier_evict oid 45 snap -1 2024-04-20T02:51:45.285 INFO:tasks.rados.rados.0.smithi105.stdout:1204: set_chunk oid 33 target oid 2024-04-20T02:51:45.288 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:45.288 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:45.300 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:45.314 INFO:tasks.rados.rados.0.smithi105.stdout:1204: set_chunk oid 33 offset: 1352192 length: 1128960 target oid 6e9f806baba5ecf3e01bd4de6e4b8cad4a50fd173079d408316bb3ca6adfd9fc offset: 0 snap -1 2024-04-20T02:51:45.314 INFO:tasks.rados.rados.0.smithi105.stdout:1205: write oid 34 current snap is 20 2024-04-20T02:51:45.314 INFO:tasks.rados.rados.0.smithi105.stdout:1202: finishing set_chunk to oid 35 2024-04-20T02:51:45.314 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 744448 length: 1264640 2024-04-20T02:51:45.314 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 124 snap 9 seq_num 124) dirty exists 2024-04-20T02:51:45.314 INFO:tasks.rados.rados.0.smithi105.stdout:1203: got (22) Invalid argument 2024-04-20T02:51:45.314 INFO:tasks.rados.rados.0.smithi105.stdout:1201: finishing write tid 1 to smithi10548823-43 2024-04-20T02:51:45.315 INFO:tasks.rados.rados.0.smithi105.stdout:1201: finishing write tid 2 to smithi10548823-43 2024-04-20T02:51:45.315 INFO:tasks.rados.rados.0.smithi105.stdout:1201: finishing write tid 3 to smithi10548823-43 2024-04-20T02:51:45.315 INFO:tasks.rados.rados.0.smithi105.stdout:1201: finishing write tid 4 to smithi10548823-43 2024-04-20T02:51:45.315 INFO:tasks.rados.rados.0.smithi105.stdout:1205: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-04-20T02:51:45.321 INFO:tasks.rados.rados.0.smithi105.stdout:1205: writing smithi10548823-34 from 796698 to 1357868 tid 1 2024-04-20T02:51:45.328 INFO:tasks.rados.rados.0.smithi105.stdout:1205: writing smithi10548823-34 from 2006323 to 2551729 tid 2 2024-04-20T02:51:45.334 INFO:tasks.rados.rados.0.smithi105.stdout:1205: writing smithi10548823-34 from 3126991 to 3596698 tid 3 2024-04-20T02:51:45.334 INFO:tasks.rados.rados.0.smithi105.stdout:1202: done (4 left) 2024-04-20T02:51:45.334 INFO:tasks.rados.rados.0.smithi105.stdout:1203: done (3 left) 2024-04-20T02:51:45.334 INFO:tasks.rados.rados.0.smithi105.stdout:1206: snap_remove snap 19 2024-04-20T02:51:45.343 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:45.343+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:45.419 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:45.420+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:45.438 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:45.438 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:45.448 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:45.532 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:45.532+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:45.616 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:45.616 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:45.621 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:45.622+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:45.626 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:45.722 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:45.722+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:45.755 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:45.756 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:45.766 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:45.823 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:45.823+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:45.923 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:45.924+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:45.934 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:45.934 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:45.944 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:46.030 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:46.030+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:46.068 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:46.116 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:46.116 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:46.125 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:46.126+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:46.127 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:46.217 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:46.230 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:46.226+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:46.232 INFO:tasks.rados.rados.0.smithi105.stdout:1206: done (3 left) 2024-04-20T02:51:46.232 INFO:tasks.rados.rados.0.smithi105.stdout:1207: read oid 38 snap -1 2024-04-20T02:51:46.232 INFO:tasks.rados.rados.0.smithi105.stdout:1207: expect deleted 2024-04-20T02:51:46.232 INFO:tasks.rados.rados.0.smithi105.stdout:1208: set_chunk oid 31 target oid 2024-04-20T02:51:46.232 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:51:46.245 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:46.266 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:46.270 INFO:tasks.rados.rados.0.smithi105.stdout:1208: set_chunk oid 31 offset: 1136640 length: 993792 target oid 3b79a8e002fc0fa93e54d4cb32783e9f2a0687a25bab0d691195d5befdea1047 offset: 0 snap -1 2024-04-20T02:51:46.270 INFO:tasks.rados.rados.0.smithi105.stdout:1209: read oid 29 snap -1 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:1209: expect (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:1210: read oid 3 snap -1 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:1210: expect (ObjNum 211 snap 20 seq_num 211) 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:1211: snap_create 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:1201: finishing write tid 5 to smithi10548823-43 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:1201: oid 43 updating version 0 to 96 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:1201: oid 43 updating version 96 to 98 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:1201: oid 43 version 98 is already newer than 97 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:1201: oid 43 version 98 is already newer than 95 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 98 (ObjNum 213 snap 20 seq_num 213) dirty exists 2024-04-20T02:51:46.271 INFO:tasks.rados.rados.0.smithi105.stdout:1201: left oid 43 (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:1204: finishing set_chunk to oid 33 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 50 (ObjNum 196 snap 19 seq_num 196) dirty exists 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:1205: finishing write tid 1 to smithi10548823-34 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:1205: finishing write tid 2 to smithi10548823-34 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:1205: finishing write tid 3 to smithi10548823-34 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:1205: finishing write tid 4 to smithi10548823-34 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:1205: finishing write tid 5 to smithi10548823-34 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:1205: oid 34 updating version 0 to 221 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:1205: oid 34 updating version 221 to 222 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:1205: oid 34 updating version 222 to 223 2024-04-20T02:51:46.272 INFO:tasks.rados.rados.0.smithi105.stdout:1205: oid 34 version 223 is already newer than 220 2024-04-20T02:51:46.273 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 223 (ObjNum 214 snap 20 seq_num 214) dirty exists 2024-04-20T02:51:46.273 INFO:tasks.rados.rados.0.smithi105.stdout:1205: left oid 34 (ObjNum 214 snap 20 seq_num 214) 2024-04-20T02:51:46.279 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:46.279 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:46.279 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:46.280 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:46.280 INFO:tasks.rados.rados.0.smithi105.stdout:1209: expect (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:51:46.290 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:46.292 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:46.310 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:46.329 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:46.329 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:46.330+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:46.361 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:46.390 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:46.391 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:46.391 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:46.391 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:46.391 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:46.391 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:46.391 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:46.391 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:46.391 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:46.428 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:46.428+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:46.442 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:46.442 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:46.443 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:46.443 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:46.443 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:46.453 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:46.530 INFO:tasks.rados.rados.0.smithi105.stdout:1210: expect (ObjNum 211 snap 20 seq_num 211) 2024-04-20T02:51:46.530 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:46.530+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:46.624 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:46.624 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:46.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:46.624 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:46.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.624 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.038445897000003, 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:46.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:46.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:51:46.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:51:46.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:51:46.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:51:46.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:51:46.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:51:46.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:51:46.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:51:46.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.022427618999998, 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:51:46.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:51:46.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:51:46.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:51:46.637 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.637 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.637 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.02227439, 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.638 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:51:46.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:46.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.643 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.644 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:46.644 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:46.644 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.022164381000003, 2024-04-20T02:51:46.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:51:46.644 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.644 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.645 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:46.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:51:46.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:51:46.651 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.022134004999998, 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:51:46.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:46.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:46.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:51:46.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:51:46.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:51:46.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:51:46.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:46.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:51:46.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:51:46.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:51:46.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:51:46.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:51:46.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:51:46.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:51:46.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:51:46.660 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.660 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.660 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.017716585999999, 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:51:46.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:51:46.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:51:46.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:51:46.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:51:46.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:51:46.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:46.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:51:46.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:51:46.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:51:46.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:51:46.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:51:46.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:51:46.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:51:46.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:51:46.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:51:46.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:51:46.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:51:46.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:51:46.669 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.670 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.670 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.670 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:46.671 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:46.671 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.008464834999998, 2024-04-20T02:51:46.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:51:46.671 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.671 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.671 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:46.671 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:46.672 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:46.672 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:51:46.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.672 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:46.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:46.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:51:46.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:46.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:51:46.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:51:46.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:51:46.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:51:46.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:51:46.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:51:46.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:51:46.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:51:46.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:51:46.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:51:46.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:46.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:51:46.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:51:46.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:51:46.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:51:46.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:51:46.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:51:46.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:51:46.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:51:46.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:51:46.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:51:46.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:51:46.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:51:46.682 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.682 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.682 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.682 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:46.682 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.005942691999998, 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:51:46.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:51:46.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.005857825, 2024-04-20T02:51:46.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:51:46.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.005840214999999, 2024-04-20T02:51:46.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:51:46.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:51:46.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:51:46.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:51:46.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:51:46.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:51:46.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:46.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:51:46.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:51:46.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:51:46.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:51:46.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:51:46.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:51:46.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:51:46.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:51:46.701 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.701 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.701 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.702 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:991 4.b e32/22 4:d4308fef:::smithi10548823-6:head v 32'29, mlcod=30'27)", 2024-04-20T02:51:46.702 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:46.702 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.898935960999999, 2024-04-20T02:51:46.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046400134000000003, 2024-04-20T02:51:46.702 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.702 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.703 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:46.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:46.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724649+0000", 2024-04-20T02:51:46.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:46.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724907+0000", 2024-04-20T02:51:46.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025779699999999998 2024-04-20T02:51:46.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724924+0000", 2024-04-20T02:51:46.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7110000000000001e-05 2024-04-20T02:51:46.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724934+0000", 2024-04-20T02:51:46.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5370000000000003e-06 2024-04-20T02:51:46.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724984+0000", 2024-04-20T02:51:46.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0791000000000002e-05 2024-04-20T02:51:46.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.725011+0000", 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6299e-05 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771009+0000", 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045998144999999997 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771050+0000", 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1727999999999998e-05 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.656473607000002, 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058137664999999998, 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.711 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967112+0000", 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981895+0000", 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014783522 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981906+0000", 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0560000000000001e-05 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981912+0000", 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981956+0000", 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3455000000000003e-05 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981979+0000", 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3475000000000001e-05 2024-04-20T02:51:46.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025222+0000", 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043242622000000001 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025250+0000", 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8520000000000001e-05 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.715 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.234404808999997, 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045096171999999997, 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.389181+0000", 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401391+0000", 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012209697 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401410+0000", 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8754e-05 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.401422+0000", 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2089999999999999e-05 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.403142+0000", 2024-04-20T02:51:46.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0017199100000000001 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.403157+0000", 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.526e-05 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434233+0000", 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031076289999999999 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434278+0000", 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4478999999999999e-05 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.211465619000002, 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.205460831000003, 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.724 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:46.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:51:46.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:130 3.0 3:1b476ba0:::474a8d6c2b8981e0c13c883e0d254815cfd155e523c9b654ff4e5007d0a1fc09:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.496119823999997, 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045176642000000003, 2024-04-20T02:51:46.728 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128171+0000", 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070576600000000003 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128183+0000", 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1542999999999999e-05 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128191+0000", 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5839999999999995e-06 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134592+0000", 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006400233 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134784+0000", 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000191831 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135041+0000", 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000257123 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162360+0000", 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027319774000000002 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162476+0000", 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115505 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172550+0000", 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01007366 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172598+0000", 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8216000000000001e-05 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172643+0000", 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5305000000000001e-05 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:135 3.0 3:12659911:::ad802ca609ac9a6b9a438bdacacdb4fb27f17cd228f3ff5506eb31f643d749ff:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.491091867999998, 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045928208999999998, 2024-04-20T02:51:46.734 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132493+0000", 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133291+0000", 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079715799999999996 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133301+0000", 2024-04-20T02:51:46.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0533000000000001e-05 2024-04-20T02:51:46.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133309+0000", 2024-04-20T02:51:46.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9910000000000002e-06 2024-04-20T02:51:46.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151891+0000", 2024-04-20T02:51:46.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018582155 2024-04-20T02:51:46.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152003+0000", 2024-04-20T02:51:46.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112064 2024-04-20T02:51:46.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:46.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152211+0000", 2024-04-20T02:51:46.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020773100000000001 2024-04-20T02:51:46.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162492+0000", 2024-04-20T02:51:46.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010281047999999999 2024-04-20T02:51:46.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169484+0000", 2024-04-20T02:51:46.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006991929 2024-04-20T02:51:46.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178312+0000", 2024-04-20T02:51:46.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0088282789999999996 2024-04-20T02:51:46.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178374+0000", 2024-04-20T02:51:46.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2123000000000001e-05 2024-04-20T02:51:46.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178423+0000", 2024-04-20T02:51:46.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201999999999997e-05 2024-04-20T02:51:46.742 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.742 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.742 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.742 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:138 3.0 3:2a051c65:::f4051f504bf0c69c0b0fcc37af36dc6836e32eef2351291df998149766aaa9b9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:46.742 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:46.743 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.490270451000001, 2024-04-20T02:51:46.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045257957000000001, 2024-04-20T02:51:46.743 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.743 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.743 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:46.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:46.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133315+0000", 2024-04-20T02:51:46.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:46.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133606+0000", 2024-04-20T02:51:46.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029134199999999999 2024-04-20T02:51:46.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133615+0000", 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2479999999999996e-06 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133621+0000", 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7739999999999999e-06 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159641+0000", 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026020008000000001 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159813+0000", 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017143200000000001 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160027+0000", 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021374400000000001 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:46.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.163217+0000", 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031905869999999999 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169517+0000", 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063002290000000001 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178484+0000", 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0089668070000000002 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178534+0000", 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9654000000000003e-05 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178574+0000", 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9929999999999999e-05 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.534952701000002, 2024-04-20T02:51:46.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049605908999999997, 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088632+0000", 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090096+0000", 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001463675 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090112+0000", 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5557999999999999e-05 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090125+0000", 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3794999999999999e-05 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090196+0000", 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0537000000000006e-05 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090224+0000", 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8555999999999999e-05 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138207+0000", 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047982536999999999 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138240+0000", 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2480000000000001e-05 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:46.753 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.482244708, 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056137746000000002, 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:46.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141340+0000", 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142751+0000", 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014113540000000001 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142770+0000", 2024-04-20T02:51:46.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8450000000000001e-05 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142780+0000", 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0284e-05 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142982+0000", 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000202244 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.143007+0000", 2024-04-20T02:51:46.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4929000000000001e-05 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197446+0000", 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054438519999999997 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197479+0000", 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3597999999999999e-05 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:46.757 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:46.758 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:46.759 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:46.760 INFO:tasks.rados.rados.0.smithi105.stdout:1208: finishing set_chunk to oid 31 2024-04-20T02:51:46.760 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 64 (ObjNum 142 snap 9 seq_num 142) dirty exists 2024-04-20T02:51:46.760 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:46.759+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:46.860 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:46.860+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:46.931 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:46.932 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:46.932 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:46.932 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:46.942 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:46.986 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:46.986+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:47.061 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:47.062+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:47.070 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:47.070 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:47.070 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:47.081 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:47.166 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:47.166+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1201: done (7 left) 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1204: done (6 left) 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1205: done (5 left) 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1207: done (4 left) 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1208: done (3 left) 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1209: done (2 left) 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1210: done (1 left) 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1211: done (0 left) 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1212: read oid 19 snap -1 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1212: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1213: read oid 39 snap -1 2024-04-20T02:51:47.235 INFO:tasks.rados.rados.0.smithi105.stdout:1213: expect (ObjNum 182 snap 15 seq_num 182) 2024-04-20T02:51:47.236 INFO:tasks.rados.rados.0.smithi105.stdout:1214: read oid 46 snap -1 2024-04-20T02:51:47.236 INFO:tasks.rados.rados.0.smithi105.stdout:1214: expect (ObjNum 192 snap 19 seq_num 192) 2024-04-20T02:51:47.236 INFO:tasks.rados.rados.0.smithi105.stdout:1215: write (excl) oid 24 current snap is 21 2024-04-20T02:51:47.236 INFO:tasks.rados.rados.0.smithi105.stdout:1215: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-04-20T02:51:47.241 INFO:tasks.rados.rados.0.smithi105.stdout:1215: writing smithi10548823-24 from 413505 to 1049924 tid 1 2024-04-20T02:51:47.257 INFO:tasks.rados.rados.0.smithi105.stdout:1215: writing smithi10548823-24 from 1664805 to 2353869 tid 2 2024-04-20T02:51:47.259 INFO:tasks.rados.rados.0.smithi105.stdout:1215: writing smithi10548823-24 from 3038061 to 3530132 tid 3 2024-04-20T02:51:47.259 INFO:tasks.rados.rados.0.smithi105.stdout:1216: set_chunk oid 14 target oid 2024-04-20T02:51:47.263 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:47.262+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:47.264 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:47.264 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:47.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:47.264 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:47.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.264 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:47.264 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.318890903000003, 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041399200999999997, 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945227+0000", 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947429+0000", 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022028989999999999 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947448+0000", 2024-04-20T02:51:47.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.821e-05 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947460+0000", 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2373e-05 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947511+0000", 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0844999999999999e-05 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947531+0000", 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0017000000000002e-05 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986580+0000", 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039049213999999999 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986627+0000", 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7070000000000002e-05 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1145 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'30, mlcod=34'26)", 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.300160204000001, 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026752594000000001, 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963958+0000", 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964662+0000", 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070467299999999995 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964678+0000", 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5597999999999999e-05 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964687+0000", 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2979999999999997e-06 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968184+0000", 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034972250000000001 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968211+0000", 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6829999999999999e-05 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990640+0000", 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022429317000000001 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990711+0000", 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0794e-05 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.272 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:129 3.0 e42/21 3:0256f537:::3f82a762bf2204de901e10565f70e759ee2a26bf55ffef4c8d5027a47e96ec31:head v 42'78, mlcod=42'76)", 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.129471629000001, 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024418002000000001, 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134646+0000", 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012265099999999999 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134789+0000", 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9924000000000001e-05 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134799+0000", 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0042e-05 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134842+0000", 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3791000000000001e-05 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134866+0000", 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3791e-05 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159017+0000", 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024150919 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159065+0000", 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201000000000001e-05 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.276 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:130 3.0 e42/21 3:1b476ba0:::474a8d6c2b8981e0c13c883e0d254815cfd155e523c9b654ff4e5007d0a1fc09:head v 42'79, mlcod=42'76)", 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.128942017, 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023928118000000002, 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:47.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135177+0000", 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135176+0000", 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135295+0000", 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118847 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135304+0000", 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4280000000000002e-06 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135311+0000", 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4810000000000003e-06 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141973+0000", 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066622219999999998 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.142009+0000", 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5725000000000001e-05 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159073+0000", 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017064019999999999 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159105+0000", 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2081e-05 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:134 3.0 e42/21 3:33aaac3f:::d0aca4dab23a9f22fb58211e350c7c1c1b7601bafbb496b93afbbc91e2ed921c:head v 42'81, mlcod=42'76)", 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.119449430000003, 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024591152000000002, 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144949+0000", 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028058500000000002 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144959+0000", 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0335e-05 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144967+0000", 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6040000000000004e-06 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151637+0000", 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066694670000000001 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151659+0000", 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2855000000000001e-05 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169224+0000", 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017565041 2024-04-20T02:51:47.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169261+0000", 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6179e-05 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1462 3.0 e44/21 3:29b349f1:::ea1112aea5e7be379214f0429a6e898cf4f4a05bdbb29b6cd15764b4776418b1:head v 44'85, mlcod=42'83)", 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.086329544999998, 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034060275000000001, 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.285 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:47.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178932+0000", 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011426069999999999 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178944+0000", 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.287e-05 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178962+0000", 2024-04-20T02:51:47.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7669999999999999e-05 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179013+0000", 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1295000000000003e-05 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179038+0000", 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4859999999999999e-05 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211813+0000", 2024-04-20T02:51:47.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032775190000000003 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211850+0000", 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6216e-05 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1629 3.0 e51/21 3:088cc2f3:::a1744cc895df5e9d3e624b281fcb9edc57f3f342d1f108d4c38dc778f219a41a:head v 51'93, mlcod=45'91)", 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.461030207999997, 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035323188999999998, 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.289 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803088+0000", 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804187+0000", 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00109891 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804201+0000", 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3772e-05 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804213+0000", 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2335e-05 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804285+0000", 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1271000000000003e-05 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804306+0000", 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1359999999999999e-05 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838377+0000", 2024-04-20T02:51:47.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034070994 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838412+0000", 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4790999999999999e-05 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1653 4.6 e51/22 4:697da67d:::smithi10548823-48:head v 51'41, mlcod=45'37)", 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.839291099999997, 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037653640000000002, 2024-04-20T02:51:47.293 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424827+0000", 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425066+0000", 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023957399999999999 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425117+0000", 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1082999999999999e-05 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425146+0000", 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8129000000000001e-05 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425217+0000", 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1402000000000005e-05 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425278+0000", 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1498e-05 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462456+0000", 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037177106000000001 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462481+0000", 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5851e-05 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1786 4.d e56/22 4:b376c20e:::smithi10548823-49:head v 56'80, mlcod=55'77)", 2024-04-20T02:51:47.297 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.452199563000001, 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028136330000000001, 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812644+0000", 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072510100000000002 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812656+0000", 2024-04-20T02:51:47.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1688999999999999e-05 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812663+0000", 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812707+0000", 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4573999999999997e-05 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812722+0000", 2024-04-20T02:51:47.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5319e-05 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840021+0000", 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027298111999999999 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840056+0000", 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5086000000000003e-05 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.301 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1799 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'53, mlcod=54'49)", 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.189870542000001, 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050527214000000001, 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074614+0000", 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036631499999999998 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074624+0000", 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8330000000000006e-06 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074632+0000", 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3599999999999998e-06 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074683+0000", 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1137999999999997e-05 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074704+0000", 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1616000000000001e-05 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124714+0000", 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050009642 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124776+0000", 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1581000000000006e-05 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.305 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.175374445999999, 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046184680999999998, 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088743+0000", 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090327+0000", 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001584071 2024-04-20T02:51:47.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090343+0000", 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5193e-05 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090354+0000", 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1336e-05 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090411+0000", 2024-04-20T02:51:47.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6589999999999999e-05 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090431+0000", 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0579000000000001e-05 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134883+0000", 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044451877000000001 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134929+0000", 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6131000000000003e-05 2024-04-20T02:51:47.309 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1800 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'54, mlcod=54'49)", 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.162968372000002, 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027029681999999999, 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:47.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101150+0000", 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101284+0000", 2024-04-20T02:51:47.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000134688 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101295+0000", 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101305+0000", 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0930999999999999e-05 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101347+0000", 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1522e-05 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101369+0000", 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2027000000000001e-05 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128138+0000", 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026768626 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128180+0000", 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2568999999999998e-05 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1807 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'87, mlcod=56'82)", 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.155287545999997, 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026347280000000001, 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108830+0000", 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109221+0000", 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039087699999999998 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109239+0000", 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8904000000000001e-05 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109271+0000", 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1319000000000003e-05 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120423+0000", 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011151912999999999 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120454+0000", 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1464999999999998e-05 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135158+0000", 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014703554000000001 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135179+0000", 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0965999999999999e-05 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2038 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'159, mlcod=0'0)", 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.682498973000001, 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044306238999999997, 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.318 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581964+0000", 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034420099999999998 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581980+0000", 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6578999999999998e-05 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581991+0000", 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0949e-05 2024-04-20T02:51:47.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582734+0000", 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074277299999999996 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582761+0000", 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6627000000000001e-05 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625882+0000", 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043121617000000001 2024-04-20T02:51:47.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625926+0000", 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3838999999999998e-05 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2039 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'160, mlcod=0'0)", 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.662100888000001, 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027318719000000002, 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.322 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602017+0000", 2024-04-20T02:51:47.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:47.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602364+0000", 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000347139 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602375+0000", 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.128e-05 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602384+0000", 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2600000000000005e-06 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605223+0000", 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0028393400000000001 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605245+0000", 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2078000000000001e-05 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629297+0000", 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024051605 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629337+0000", 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0126000000000002e-05 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2137 3.0 e70/21 3:0d26903d:::7146376437549e0b99264d159bf1e9c8bc43ac1c7eb519f65899cfd592a43a4a:head v 70'98, mlcod=55'96)", 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.479972904, 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024394852000000002, 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784144+0000", 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785280+0000", 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001135741 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785299+0000", 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8845999999999999e-05 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785310+0000", 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1450999999999999e-05 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785362+0000", 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1798000000000001e-05 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785388+0000", 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5763999999999999e-05 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808507+0000", 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023118742000000001 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808541+0000", 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4258000000000003e-05 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2142 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'65, mlcod=66'61)", 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.447273491000001, 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033365672999999998, 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816981+0000", 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135952 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816994+0000", 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3405e-05 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817003+0000", 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4319999999999999e-06 2024-04-20T02:51:47.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817055+0000", 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1842999999999998e-05 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817076+0000", 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1375000000000001e-05 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850179+0000", 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033103051000000001 2024-04-20T02:51:47.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850211+0000", 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1896000000000001e-05 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2143 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'66, mlcod=66'61)", 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 19.433407648999999, 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023949208999999999, 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.336 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830710+0000", 2024-04-20T02:51:47.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830883+0000", 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017331000000000001 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830903+0000", 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0043000000000001e-05 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830914+0000", 2024-04-20T02:51:47.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1263999999999999e-05 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.836558+0000", 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056437309999999999 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.836587+0000", 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.854633+0000", 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01804584 2024-04-20T02:51:47.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.854660+0000", 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7708e-05 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.122738876, 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039421115999999999, 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.340 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.341 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:47.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:47.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141379+0000", 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142797+0000", 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001418702 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142816+0000", 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8907999999999999e-05 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142825+0000", 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1020000000000003e-06 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142872+0000", 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6168999999999998e-05 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142891+0000", 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9561000000000001e-05 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180704+0000", 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037812643 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180801+0000", 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7361999999999994e-05 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2280 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'105, mlcod=73'100)", 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.102057599, 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029657706999999998, 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162060+0000", 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162455+0000", 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039569799999999999 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162472+0000", 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6820000000000002e-05 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162485+0000", 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2388999999999999e-05 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163758+0000", 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001273906 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163784+0000", 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5352000000000001e-05 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191663+0000", 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027879583999999999 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191719+0000", 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640000000000003e-05 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.348 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.349 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.349 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.349 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:47.350 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:47.351 INFO:tasks.rados.rados.0.smithi105.stdout:1216: set_chunk oid 14 offset: 625152 length: 491520 target oid 104de298d1ee51baed4f73f04496f3d6c57fc184ab365b1aa735ea91d7a37b80 offset: 0 snap -1 2024-04-20T02:51:47.351 INFO:tasks.rados.rados.0.smithi105.stdout:1217: tier_flush oid 37 2024-04-20T02:51:47.351 INFO:tasks.rados.rados.0.smithi105.stdout:1217: tier_flush oid 37 snap -1 2024-04-20T02:51:47.351 INFO:tasks.rados.rados.0.smithi105.stdout:1213: expect (ObjNum 182 snap 15 seq_num 182) 2024-04-20T02:51:47.391 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:47.391+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:47.422 INFO:tasks.rados.rados.0.smithi105.stdout:1214: expect (ObjNum 192 snap 19 seq_num 192) 2024-04-20T02:51:47.464 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:47.464+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:47.522 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:47.522 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:47.522 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:47.523 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:47.530 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:47.561 INFO:tasks.rados.rados.0.smithi105.stdout:1212: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:47.605 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:47.605+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:47.666 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:47.667+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:47.680 INFO:tasks.rados.rados.0.smithi105.stdout:1215: finishing write tid 1 to smithi10548823-24 2024-04-20T02:51:47.680 INFO:tasks.rados.rados.0.smithi105.stdout:1215: finishing write tid 2 to smithi10548823-24 2024-04-20T02:51:47.680 INFO:tasks.rados.rados.0.smithi105.stdout:1215: finishing write tid 3 to smithi10548823-24 2024-04-20T02:51:47.680 INFO:tasks.rados.rados.0.smithi105.stdout:1215: finishing write tid 4 to smithi10548823-24 2024-04-20T02:51:47.680 INFO:tasks.rados.rados.0.smithi105.stdout:1212: done (5 left) 2024-04-20T02:51:47.680 INFO:tasks.rados.rados.0.smithi105.stdout:1213: done (4 left) 2024-04-20T02:51:47.680 INFO:tasks.rados.rados.0.smithi105.stdout:1214: done (3 left) 2024-04-20T02:51:47.680 INFO:tasks.rados.rados.0.smithi105.stdout:1218: read oid 35 snap -1 2024-04-20T02:51:47.680 INFO:tasks.rados.rados.0.smithi105.stdout:1218: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:51:47.680 INFO:tasks.rados.rados.0.smithi105.stdout:1219: set_chunk oid 4 target oid 2024-04-20T02:51:47.695 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:47.695 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:47.697 INFO:tasks.rados.rados.0.smithi105.stdout:1219: set_chunk oid 4 offset: 2541056 length: 521216 target oid 5600146fae0b9eb4609c7df3e5657ddfc04b45f4bcc52f496302d841c0e2b4fc offset: 0 snap -1 2024-04-20T02:51:47.697 INFO:tasks.rados.rados.0.smithi105.stdout:1220: snap_remove snap 20 2024-04-20T02:51:47.706 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:47.767 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:47.767+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:47.868 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:47.868+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:47.875 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:47.875 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:47.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 61.331131476000003, 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:51:47.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:51:47.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:47.881 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 61.324910402, 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:47.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:51:47.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 61.316510198000003, 2024-04-20T02:51:47.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:51:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:51:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:51:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:51:47.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:51:47.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.893 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:971 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (56) in=7b,truncate 3277332] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 61.316192129000001, 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051333776999999997, 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.894 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 971 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558678+0000", 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558686+0000", 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7330000000000003e-06 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558693+0000", 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.866e-06 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558698+0000", 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.49e-06 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594685+0000", 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035986753000000003 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594792+0000", 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107129 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594958+0000", 2024-04-20T02:51:47.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000165429 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595721+0000", 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076360100000000004 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.596242+0000", 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052083800000000003 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609915+0000", 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013673431 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609958+0000", 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2759000000000002e-05 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610013+0000", 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4421e-05 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.899 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 61.316167553, 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:51:47.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.904 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 53.509845550000001, 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 53.502582588000003, 2024-04-20T02:51:47.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:51:47.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:51:47.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:51:47.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.916 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.917 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.917 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:47.917 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:47.917 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 53.496640059000001, 2024-04-20T02:51:47.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:51:47.917 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.917 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.917 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:51:47.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:51:47.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:47.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:51:47.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:51:47.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:51:47.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:51:47.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:51:47.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:51:47.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:51:47.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:51:47.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.923 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 53.496437067000002, 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:47.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:51:47.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:47.929 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 53.496414139999999, 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:51:47.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 53.486330406, 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.934 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:51:47.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:51:47.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:51:47.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 53.469444998, 2024-04-20T02:51:47.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:47.942 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 53.453168556999998, 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:51:47.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:51:47.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.428613085000002, 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:51:47.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:51:47.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.952 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.114297309000001, 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:51:47.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.958 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2046 4.11 4:8b927dd6:::smithi10548823-28:head [write 2025476~459397] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.767100+0000", 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.107771275000001, 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050129193000000002, 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2046 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767100+0000", 2024-04-20T02:51:47.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767100+0000", 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767098+0000", 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767266+0000", 2024-04-20T02:51:47.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000167901 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767281+0000", 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5478999999999999e-05 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767294+0000", 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3259e-05 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.786324+0000", 2024-04-20T02:51:47.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019029404 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.786421+0000", 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6897000000000004e-05 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.786579+0000", 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015888299999999999 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804648+0000", 2024-04-20T02:51:47.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018068208999999998 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804685+0000", 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7832e-05 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817139+0000", 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012453862 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817184+0000", 2024-04-20T02:51:47.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4437000000000002e-05 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817229+0000", 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5136000000000003e-05 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2047 4.11 4:8b927dd6:::smithi10548823-28:head [write 2971091~70976] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.770122+0000", 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.104748920999999, 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.048007332999999999, 2024-04-20T02:51:47.964 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2047 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770122+0000", 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770122+0000", 2024-04-20T02:51:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770120+0000", 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770185+0000", 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4204999999999999e-05 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770199+0000", 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4362000000000001e-05 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770212+0000", 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2649999999999999e-05 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804711+0000", 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034499582000000001 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804818+0000", 2024-04-20T02:51:47.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107203 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804983+0000", 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016515200000000001 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817237+0000", 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012254034 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817905+0000", 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00066738500000000005 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818036+0000", 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013161999999999999 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818081+0000", 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4734999999999998e-05 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818129+0000", 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8167999999999997e-05 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2048 4.11 4:8b927dd6:::smithi10548823-28:head [setxattr _header (57) in=7b,truncate 3042067] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.770221+0000", 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.104650320000001, 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049037753000000003, 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.970 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2048 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770221+0000", 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770221+0000", 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770220+0000", 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770223+0000", 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.32e-06 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770230+0000", 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2759999999999997e-06 2024-04-20T02:51:47.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770238+0000", 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2539999999999992e-06 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816299+0000", 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046060940000000002 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816426+0000", 2024-04-20T02:51:47.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012759199999999999 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816612+0000", 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000185885 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818165+0000", 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001552138 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818192+0000", 2024-04-20T02:51:47.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7271e-05 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819143+0000", 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00095093499999999997 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819199+0000", 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6334e-05 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819258+0000", 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9373000000000002e-05 2024-04-20T02:51:47.975 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2049 4.11 4:8b927dd6:::smithi10548823-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.104628534, 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049230470999999998, 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2049 2024-04-20T02:51:47.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:47.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770245+0000", 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4430000000000002e-06 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770251+0000", 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2929999999999997e-06 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770260+0000", 2024-04-20T02:51:47.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2930000000000006e-06 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816910+0000", 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046650129999999998 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817004+0000", 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3665999999999994e-05 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819289+0000", 2024-04-20T02:51:47.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002284847 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819379+0000", 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0274999999999999e-05 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819473+0000", 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3882999999999995e-05 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2050 4.0 4:007b2a87:::smithi10548823-39:head [write 658874~771437,stat] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.778080+0000", 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.096791460999999, 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051190877000000003, 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2050 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:47.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778080+0000", 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778080+0000", 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778077+0000", 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778477+0000", 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039974900000000002 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778496+0000", 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9335e-05 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778510+0000", 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3377000000000001e-05 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778572+0000", 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1898999999999998e-05 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778685+0000", 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011372800000000001 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778917+0000", 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195799999999999 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817646+0000", 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038729087000000002 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817828+0000", 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181986 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.829157+0000", 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011328379 2024-04-20T02:51:47.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.829216+0000", 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8962000000000002e-05 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.829270+0000", 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4727e-05 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:47.986 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:47.988 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:47.988 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:47.988+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:48.089 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:48.089+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:48.130 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:48.131 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:48.131 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:48.131 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:48.141 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:48.226 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:48.227+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:48.241 INFO:tasks.rados.rados.0.smithi105.stdout:1220: done (5 left) 2024-04-20T02:51:48.241 INFO:tasks.rados.rados.0.smithi105.stdout:1221: set_chunk oid 7 target oid 2024-04-20T02:51:48.250 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:51:48.271 INFO:tasks.rados.rados.0.smithi105.stdout:1221: set_chunk oid 7 offset: 1859072 length: 1004544 target oid 2dd0379af219fdb9b8cadbe90fdf3f0103964d17d43eb7eeb833e0df5583345a offset: 0 snap -1 2024-04-20T02:51:48.271 INFO:tasks.rados.rados.0.smithi105.stdout:1222: set_chunk oid 18 target oid 2024-04-20T02:51:48.271 INFO:tasks.rados.rados.0.smithi105.stdout:1222: set_chunk oid 18 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:51:48.271 INFO:tasks.rados.rados.0.smithi105.stdout:1223: read oid 12 snap -1 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1223: expect (ObjNum 167 snap 14 seq_num 167) 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1215: finishing write tid 5 to smithi10548823-24 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1215: oid 24 updating version 0 to 246 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1215: oid 24 updating version 246 to 247 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1215: oid 24 updating version 247 to 249 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1215: oid 24 version 249 is already newer than 248 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 249 (ObjNum 215 snap 21 seq_num 215) dirty exists 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1215: left oid 24 (ObjNum 215 snap 21 seq_num 215) 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1216: finishing set_chunk to oid 14 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 41 (ObjNum 99 snap 5 seq_num 99) dirty exists 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1217: got (0) Success 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 0 (ObjNum 184 snap 17 seq_num 184) dirty exists 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1224: read oid 26 snap -1 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1224: expect deleted 2024-04-20T02:51:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1215: done (8 left) 2024-04-20T02:51:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1216: done (7 left) 2024-04-20T02:51:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1217: done (6 left) 2024-04-20T02:51:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1225: tier_promote oid 5 2024-04-20T02:51:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1226: read oid 29 snap -1 2024-04-20T02:51:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1226: expect (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:51:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1227: read oid 31 snap -1 2024-04-20T02:51:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1227: expect (ObjNum 142 snap 9 seq_num 142) 2024-04-20T02:51:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1228: write (excl) oid 17 current snap is 21 2024-04-20T02:51:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1228: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-04-20T02:51:48.278 INFO:tasks.rados.rados.0.smithi105.stdout:1228: writing smithi10548823-17 from 430312 to 1141980 tid 1 2024-04-20T02:51:48.284 INFO:tasks.rados.rados.0.smithi105.stdout:1228: writing smithi10548823-17 from 1723287 to 2272362 tid 2 2024-04-20T02:51:48.290 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:48.291+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:48.291 INFO:tasks.rados.rados.0.smithi105.stdout:1228: writing smithi10548823-17 from 2781837 to 3485180 tid 3 2024-04-20T02:51:48.291 INFO:tasks.rados.rados.0.smithi105.stdout:1229: delete oid 15 current snap is 21 2024-04-20T02:51:48.292 INFO:tasks.rados.rados.0.smithi105.stdout:1218: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:51:48.329 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:48.329 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:48.329 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:48.329 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:48.329 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:48.340 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:48.363 INFO:tasks.rados.rados.0.smithi105.stdout:1219: finishing set_chunk to oid 4 2024-04-20T02:51:48.363 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 2541056 length: 521216 2024-04-20T02:51:48.363 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 191 snap 19 seq_num 191) dirty exists 2024-04-20T02:51:48.363 INFO:tasks.rados.rados.0.smithi105.stdout:1218: done (10 left) 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1219: done (9 left) 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1229: done (8 left) 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1230: tier_promote oid 34 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1222: finishing set_chunk to oid 18 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1222: got expected ENOENT (src dne) 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1225: got (0) Success 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 116 (ObjNum 186 snap 18 seq_num 186) dirty exists 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1221: finishing set_chunk to oid 7 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 167 (ObjNum 199 snap 19 seq_num 199) dirty exists 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1221: done (8 left) 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1222: done (7 left) 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1224: done (6 left) 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1225: done (5 left) 2024-04-20T02:51:48.364 INFO:tasks.rados.rados.0.smithi105.stdout:1231: read oid 3 snap -1 2024-04-20T02:51:48.365 INFO:tasks.rados.rados.0.smithi105.stdout:1231: expect (ObjNum 211 snap 20 seq_num 211) 2024-04-20T02:51:48.365 INFO:tasks.rados.rados.0.smithi105.stdout:1232: set_chunk oid 11 target oid 2024-04-20T02:51:48.389 INFO:tasks.rados.rados.0.smithi105.stdout:1232: set_chunk oid 11 offset: 1623040 length: 993792 target oid 315a981e9097f33e94eac61f9146a3a6147d149b176d78b86922f59fdb34d661 offset: 0 snap -1 2024-04-20T02:51:48.389 INFO:tasks.rados.rados.0.smithi105.stdout:1233: set_chunk oid 39 target oid 2024-04-20T02:51:48.404 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:48.407 INFO:tasks.rados.rados.0.smithi105.stdout:1233: set_chunk oid 39 offset: 2003968 length: 505856 target oid e1e80d3d4d1631283babf4d786be4570772d5939e9560ec35585a0ae2fae5272 offset: 0 snap -1 2024-04-20T02:51:48.407 INFO:tasks.rados.rados.0.smithi105.stdout:1234: read oid 26 snap -1 2024-04-20T02:51:48.408 INFO:tasks.rados.rados.0.smithi105.stdout:1234: expect deleted 2024-04-20T02:51:48.408 INFO:tasks.rados.rados.0.smithi105.stdout:1235: read oid 41 snap -1 2024-04-20T02:51:48.408 INFO:tasks.rados.rados.0.smithi105.stdout:1235: expect deleted 2024-04-20T02:51:48.408 INFO:tasks.rados.rados.0.smithi105.stdout:1236: write oid 40 current snap is 21 2024-04-20T02:51:48.408 INFO:tasks.rados.rados.0.smithi105.stdout:1236: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-04-20T02:51:48.412 INFO:tasks.rados.rados.0.smithi105.stdout:1236: writing smithi10548823-40 from 463926 to 926092 tid 1 2024-04-20T02:51:48.420 INFO:tasks.rados.rados.0.smithi105.stdout:1236: writing smithi10548823-40 from 1723898 to 2160289 tid 2 2024-04-20T02:51:48.424 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:48.425+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:48.428 INFO:tasks.rados.rados.0.smithi105.stdout:1236: writing smithi10548823-40 from 2603977 to 3329864 tid 3 2024-04-20T02:51:48.428 INFO:tasks.rados.rados.0.smithi105.stdout:1237: rollback oid 43 current snap is 21 2024-04-20T02:51:48.428 INFO:tasks.rados.rados.0.smithi105.stdout:1237: done (11 left) 2024-04-20T02:51:48.428 INFO:tasks.rados.rados.0.smithi105.stdout:1238: write (excl) oid 7 current snap is 21 2024-04-20T02:51:48.428 INFO:tasks.rados.rados.0.smithi105.stdout:1238: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-04-20T02:51:48.433 INFO:tasks.rados.rados.0.smithi105.stdout:1238: writing smithi10548823-7 from 480733 to 1018148 tid 1 2024-04-20T02:51:48.443 INFO:tasks.rados.rados.0.smithi105.stdout:1238: writing smithi10548823-7 from 1782380 to 2478782 tid 2 2024-04-20T02:51:48.450 INFO:tasks.rados.rados.0.smithi105.stdout:1238: writing smithi10548823-7 from 3031400 to 3568559 tid 3 2024-04-20T02:51:48.450 INFO:tasks.rados.rados.0.smithi105.stdout:1239: read oid 43 snap -1 2024-04-20T02:51:48.451 INFO:tasks.rados.rados.0.smithi105.stdout:1239: expect (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:51:48.451 INFO:tasks.rados.rados.0.smithi105.stdout:1240: set_chunk oid 50 target oid 2024-04-20T02:51:48.492 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:48.492+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:48.508 INFO:tasks.rados.rados.0.smithi105.stdout:1240: set_chunk oid 50 offset: 727552 length: 1677312 target oid 348e4a2a55868b09db2e6d173951b4676cfef7376c5a81f7a82bb8f7ccc4ada5 offset: 0 snap -1 2024-04-20T02:51:48.508 INFO:tasks.rados.rados.0.smithi105.stdout:1241: set_chunk oid 21 target oid 2024-04-20T02:51:48.530 INFO:tasks.rados.rados.0.smithi105.stdout:1241: set_chunk oid 21 offset: 849920 length: 1069056 target oid 2ce1151df74dc7e245171c4c3aff6a773e9c662f2197aea40619905e2111c059 offset: 0 snap -1 2024-04-20T02:51:48.530 INFO:tasks.rados.rados.0.smithi105.stdout:1242: read oid 49 snap -1 2024-04-20T02:51:48.530 INFO:tasks.rados.rados.0.smithi105.stdout:1242: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:48.530 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:51:48.530 INFO:tasks.rados.rados.0.smithi105.stdout:1226: expect (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.598006466000001, 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.542 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:48.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:48.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:51:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:51:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:51:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:51:48.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:51:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:51:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:51:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:51:48.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.590528147999997, 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.550 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:51:48.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.556 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1146 4.d 4:b376c20e:::smithi10548823-49:head [write 3437750~125301] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.585960286999999, 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070707487999999999, 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1146 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955022+0000", 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955179+0000", 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015657 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955188+0000", 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.645e-06 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955195+0000", 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2910000000000004e-06 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978329+0000", 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02313401 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978450+0000", 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121438 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978652+0000", 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020163700000000001 2024-04-20T02:51:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984380+0000", 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057282510000000002 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991325+0000", 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069446830000000001 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025680+0000", 2024-04-20T02:51:48.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034355284 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025709+0000", 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8781000000000001e-05 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025731+0000", 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1963e-05 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1147 4.d 4:b376c20e:::smithi10548823-49:head [setxattr _header (56) in=7b,truncate 3563051] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.585781949999998, 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070713812000000001, 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1147 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955201+0000", 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955204+0000", 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.05e-06 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955207+0000", 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1949999999999999e-06 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955210+0000", 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983351+0000", 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028140978000000001 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983488+0000", 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013647 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983668+0000", 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000180335 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984487+0000", 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081847099999999998 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991393+0000", 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069068569999999998 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025852+0000", 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034458946999999997 2024-04-20T02:51:48.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025883+0000", 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0261e-05 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025915+0000", 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2654999999999998e-05 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.568 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.585770248999999, 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:51:48.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.573 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 54.162341314999999, 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:51:48.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:51:48.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 54.162304431999999, 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:51:48.580 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.586 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 54.162279798999997, 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.587 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:51:48.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:51:48.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:51:48.591 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 54.155786388000003, 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:51:48.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:51:48.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:51:48.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:51:48.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:51:48.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 54.149722365999999, 2024-04-20T02:51:48.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:51:48.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:51:48.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 54.139660601000003, 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:51:48.604 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 54.138967997000002, 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.610 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:51:48.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:51:48.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 54.138956342999997, 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:48.616 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:51:48.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:51:48.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 48.432891787000003, 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.621 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1426 4.1 4:8b927dd6:::smithi10548823-28:head [write 1331027~582033] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 48.421337625, 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068185170000000003, 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.625 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1426 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:48.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119645+0000", 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120235+0000", 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059073600000000002 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120249+0000", 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4168000000000001e-05 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120258+0000", 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.2719999999999997e-06 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124047+0000", 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003788839 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124154+0000", 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000107747 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124307+0000", 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000152998 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148497+0000", 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024189789999999999 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168007+0000", 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019509463000000001 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187765+0000", 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019758738000000001 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187802+0000", 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7153999999999997e-05 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187831+0000", 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8592999999999999e-05 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1428 4.1 4:8b927dd6:::smithi10548823-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 48.420684796000003, 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067836666000000004, 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:48.631 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1428 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:48.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120298+0000", 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120301+0000", 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.853e-06 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120308+0000", 2024-04-20T02:51:48.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0559999999999999e-06 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120313+0000", 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1680000000000003e-06 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148583+0000", 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028269882 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:48.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148673+0000", 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9922999999999999e-05 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187934+0000", 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039260647000000003 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188031+0000", 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7468999999999998e-05 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188135+0000", 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104126 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 48.413831070999997, 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:51:48.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:51:48.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:51:48.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:51:48.641 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 48.407884756999998, 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.642 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.647 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 48.407319315999999, 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.648 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:48.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:51:48.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:51:48.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:51:48.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 48.407293285000002, 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:48.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:48.658 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:48.660 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:48.660 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:48.660 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:48.660 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:48.660 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:48.661 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:48.661 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:48.660+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:48.676 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:48.695 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:48.715 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:48.716 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:48.716 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:48.716 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:48.716 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:48.717 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:48.717 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:48.717 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:48.717 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:48.760 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:48.761+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:48.772 INFO:tasks.rados.rados.0.smithi105.stdout:1223: expect (ObjNum 167 snap 14 seq_num 167) 2024-04-20T02:51:48.829 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:48.829 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:48.839 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:48.880 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:48.881+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:48.914 INFO:tasks.rados.rados.0.smithi105.stdout:1227: expect (ObjNum 142 snap 9 seq_num 142) 2024-04-20T02:51:48.962 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:48.962+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:49.007 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:49.008 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:49.018 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:49.046 INFO:tasks.rados.rados.0.smithi105.stdout:1228: finishing write tid 1 to smithi10548823-17 2024-04-20T02:51:49.046 INFO:tasks.rados.rados.0.smithi105.stdout:1228: finishing write tid 2 to smithi10548823-17 2024-04-20T02:51:49.046 INFO:tasks.rados.rados.0.smithi105.stdout:1228: finishing write tid 3 to smithi10548823-17 2024-04-20T02:51:49.046 INFO:tasks.rados.rados.0.smithi105.stdout:1228: finishing write tid 4 to smithi10548823-17 2024-04-20T02:51:49.046 INFO:tasks.rados.rados.0.smithi105.stdout:1228: finishing write tid 5 to smithi10548823-17 2024-04-20T02:51:49.046 INFO:tasks.rados.rados.0.smithi105.stdout:1228: oid 17 updating version 0 to 68 2024-04-20T02:51:49.046 INFO:tasks.rados.rados.0.smithi105.stdout:1228: oid 17 updating version 68 to 70 2024-04-20T02:51:49.046 INFO:tasks.rados.rados.0.smithi105.stdout:1228: oid 17 updating version 70 to 71 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:1228: oid 17 version 71 is already newer than 69 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 71 (ObjNum 216 snap 21 seq_num 216) dirty exists 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:1228: left oid 17 (ObjNum 216 snap 21 seq_num 216) 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:1223: done (15 left) 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:1226: done (14 left) 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:1227: done (13 left) 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:1228: done (12 left) 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:1243: write (excl) oid 45 current snap is 21 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:1230: got (0) Success 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 223 (ObjNum 214 snap 20 seq_num 214) dirty exists 2024-04-20T02:51:49.047 INFO:tasks.rados.rados.0.smithi105.stdout:1231: expect (ObjNum 211 snap 20 seq_num 211) 2024-04-20T02:51:49.062 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:49.063+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:49.147 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:49.148 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:49.157 INFO:tasks.rados.rados.0.smithi105.stdout:1232: finishing set_chunk to oid 11 2024-04-20T02:51:49.157 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 88 (ObjNum 210 snap 19 seq_num 210) dirty exists 2024-04-20T02:51:49.157 INFO:tasks.rados.rados.0.smithi105.stdout:1233: finishing set_chunk to oid 39 2024-04-20T02:51:49.157 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 74 (ObjNum 182 snap 15 seq_num 182) dirty exists 2024-04-20T02:51:49.157 INFO:tasks.rados.rados.0.smithi105.stdout:1239: expect (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:51:49.158 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:49.167 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:49.164+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:49.245 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:51:49.265 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:49.265+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:49.329 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:49.329 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:49.340 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:49.382 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:49.382+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1236: finishing write tid 1 to smithi10548823-40 2024-04-20T02:51:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1236: finishing write tid 2 to smithi10548823-40 2024-04-20T02:51:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1236: finishing write tid 3 to smithi10548823-40 2024-04-20T02:51:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1236: finishing write tid 4 to smithi10548823-40 2024-04-20T02:51:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1236: finishing write tid 5 to smithi10548823-40 2024-04-20T02:51:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1236: oid 40 updating version 0 to 255 2024-04-20T02:51:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1236: oid 40 version 255 is already newer than 254 2024-04-20T02:51:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1236: oid 40 version 255 is already newer than 253 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1236: oid 40 updating version 255 to 256 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 256 (ObjNum 218 snap 21 seq_num 218) dirty exists 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1236: left oid 40 (ObjNum 218 snap 21 seq_num 218) 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1238: finishing write tid 1 to smithi10548823-7 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1238: finishing write tid 2 to smithi10548823-7 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1238: finishing write tid 3 to smithi10548823-7 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1238: finishing write tid 4 to smithi10548823-7 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1238: finishing write tid 5 to smithi10548823-7 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1238: oid 7 updating version 0 to 204 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1238: oid 7 version 204 is already newer than 203 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1238: oid 7 updating version 204 to 205 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1238: oid 7 updating version 205 to 206 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 206 (ObjNum 219 snap 21 seq_num 219) dirty exists 2024-04-20T02:51:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1238: left oid 7 (ObjNum 219 snap 21 seq_num 219) 2024-04-20T02:51:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:1240: finishing set_chunk to oid 50 2024-04-20T02:51:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 72 (ObjNum 146 snap 10 seq_num 146) dirty exists 2024-04-20T02:51:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:1241: finishing set_chunk to oid 21 2024-04-20T02:51:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk returned error code -95 offset: 849920 length: 1069056 2024-04-20T02:51:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-20T02:51:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:1243: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-04-20T02:51:49.458 INFO:tasks.rados.rados.0.smithi105.stdout:1243: writing smithi10548823-45 from 497540 to 1110204 tid 1 2024-04-20T02:51:49.464 INFO:tasks.rados.rados.0.smithi105.stdout:1243: writing smithi10548823-45 from 1840862 to 2280922 tid 2 2024-04-20T02:51:49.466 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:49.466+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1243: writing smithi10548823-45 from 3058823 to 3523607 tid 3 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1230: done (12 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1231: done (11 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1232: done (10 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1233: done (9 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1234: done (8 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1235: done (7 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1236: done (6 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1238: done (5 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1239: done (4 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1240: done (3 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1241: done (2 left) 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1244: read oid 17 snap -1 2024-04-20T02:51:49.472 INFO:tasks.rados.rados.0.smithi105.stdout:1244: expect (ObjNum 216 snap 21 seq_num 216) 2024-04-20T02:51:49.473 INFO:tasks.rados.rados.0.smithi105.stdout:1245: write (excl) oid 50 current snap is 21 2024-04-20T02:51:49.473 INFO:tasks.rados.rados.0.smithi105.stdout:1242: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:49.513 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:49.513 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:49.524 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:49.569 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:49.570+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:49.615 INFO:tasks.rados.rados.0.smithi105.stdout:1244: expect (ObjNum 216 snap 21 seq_num 216) 2024-04-20T02:51:49.653 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:49.653 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:49.663 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:49.668 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:49.669+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:49.769 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:49.769+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:49.820 INFO:tasks.rados.rados.0.smithi105.stdout:1243: finishing write tid 1 to smithi10548823-45 2024-04-20T02:51:49.820 INFO:tasks.rados.rados.0.smithi105.stdout:1243: finishing write tid 2 to smithi10548823-45 2024-04-20T02:51:49.821 INFO:tasks.rados.rados.0.smithi105.stdout:1243: finishing write tid 3 to smithi10548823-45 2024-04-20T02:51:49.821 INFO:tasks.rados.rados.0.smithi105.stdout:1243: finishing write tid 4 to smithi10548823-45 2024-04-20T02:51:49.821 INFO:tasks.rados.rados.0.smithi105.stdout:1243: finishing write tid 5 to smithi10548823-45 2024-04-20T02:51:49.821 INFO:tasks.rados.rados.0.smithi105.stdout:1243: oid 45 updating version 0 to 154 2024-04-20T02:51:49.821 INFO:tasks.rados.rados.0.smithi105.stdout:1243: oid 45 version 154 is already newer than 152 2024-04-20T02:51:49.821 INFO:tasks.rados.rados.0.smithi105.stdout:1243: oid 45 version 154 is already newer than 153 2024-04-20T02:51:49.821 INFO:tasks.rados.rados.0.smithi105.stdout:1243: oid 45 updating version 154 to 155 2024-04-20T02:51:49.821 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 155 (ObjNum 220 snap 21 seq_num 220) dirty exists 2024-04-20T02:51:49.821 INFO:tasks.rados.rados.0.smithi105.stdout:1243: left oid 45 (ObjNum 220 snap 21 seq_num 220) 2024-04-20T02:51:49.821 INFO:tasks.rados.rados.0.smithi105.stdout:1245: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-04-20T02:51:49.825 INFO:tasks.rados.rados.0.smithi105.stdout:1245: writing smithi10548823-50 from 514347 to 918613 tid 1 2024-04-20T02:51:49.832 INFO:tasks.rados.rados.0.smithi105.stdout:1245: writing smithi10548823-50 from 1499344 to 2199415 tid 2 2024-04-20T02:51:49.833 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:49.833 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:49.839 INFO:tasks.rados.rados.0.smithi105.stdout:1245: writing smithi10548823-50 from 2686246 to 3362302 tid 3 2024-04-20T02:51:49.839 INFO:tasks.rados.rados.0.smithi105.stdout:1242: done (3 left) 2024-04-20T02:51:49.839 INFO:tasks.rados.rados.0.smithi105.stdout:1243: done (2 left) 2024-04-20T02:51:49.839 INFO:tasks.rados.rados.0.smithi105.stdout:1244: done (1 left) 2024-04-20T02:51:49.839 INFO:tasks.rados.rados.0.smithi105.stdout:1246: read oid 49 snap -1 2024-04-20T02:51:49.839 INFO:tasks.rados.rados.0.smithi105.stdout:1246: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:49.839 INFO:tasks.rados.rados.0.smithi105.stdout:1247: read oid 41 snap -1 2024-04-20T02:51:49.839 INFO:tasks.rados.rados.0.smithi105.stdout:1247: expect deleted 2024-04-20T02:51:49.840 INFO:tasks.rados.rados.0.smithi105.stdout:1248: write (excl) oid 42 current snap is 21 2024-04-20T02:51:49.840 INFO:tasks.rados.rados.0.smithi105.stdout:1248: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-04-20T02:51:49.844 INFO:tasks.rados.rados.0.smithi105.stdout:1248: writing smithi10548823-42 from 531154 to 1010669 tid 1 2024-04-20T02:51:49.846 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:49.849 INFO:tasks.rados.rados.0.smithi105.stdout:1248: writing smithi10548823-42 from 1557826 to 2001555 tid 2 2024-04-20T02:51:49.856 INFO:tasks.rados.rados.0.smithi105.stdout:1248: writing smithi10548823-42 from 2713669 to 3200997 tid 3 2024-04-20T02:51:49.859 INFO:tasks.rados.rados.0.smithi105.stdout:1248: writing smithi10548823-42 from 3679295 to 3731154 tid 4 2024-04-20T02:51:49.859 INFO:tasks.rados.rados.0.smithi105.stdout:1249: set_chunk oid 13 target oid 2024-04-20T02:51:49.869 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:49.870+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:49.888 INFO:tasks.rados.rados.0.smithi105.stdout:1249: set_chunk oid 13 offset: 1805312 length: 684544 target oid e2e7277166a65ed64d46cb2087662fd24e235479d8d2b7b72029c7c1bf3a93bb offset: 0 snap -1 2024-04-20T02:51:49.888 INFO:tasks.rados.rados.0.smithi105.stdout:1250: read oid 33 snap -1 2024-04-20T02:51:49.889 INFO:tasks.rados.rados.0.smithi105.stdout:1250: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:51:49.889 INFO:tasks.rados.rados.0.smithi105.stdout:1251: tier_evict oid 16 2024-04-20T02:51:49.889 INFO:tasks.rados.rados.0.smithi105.stdout:1251: tier_evict oid 16 snap -1 2024-04-20T02:51:49.889 INFO:tasks.rados.rados.0.smithi105.stdout:1246: expect (ObjNum 152 snap 10 seq_num 152) 2024-04-20T02:51:49.970 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:49.971+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:49.984 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:49.984 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:49.995 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:50.028 INFO:tasks.rados.rados.0.smithi105.stdout:1245: finishing write tid 1 to smithi10548823-50 2024-04-20T02:51:50.028 INFO:tasks.rados.rados.0.smithi105.stdout:1245: finishing write tid 2 to smithi10548823-50 2024-04-20T02:51:50.028 INFO:tasks.rados.rados.0.smithi105.stdout:1246: done (6 left) 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1247: done (5 left) 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1252: read oid 5 snap -1 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1252: expect (ObjNum 186 snap 18 seq_num 186) 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1253: snap_create 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1251: got (2) No such file or directory 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1251: got expected ENOENT (src dne) 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1249: finishing set_chunk to oid 13 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 142 (ObjNum 151 snap 10 seq_num 151) dirty exists 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1245: finishing write tid 3 to smithi10548823-50 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1245: finishing write tid 4 to smithi10548823-50 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1245: finishing write tid 5 to smithi10548823-50 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1245: oid 50 updating version 0 to 82 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1245: oid 50 updating version 82 to 83 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1245: oid 50 updating version 83 to 84 2024-04-20T02:51:50.029 INFO:tasks.rados.rados.0.smithi105.stdout:1245: oid 50 version 84 is already newer than 81 2024-04-20T02:51:50.030 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 84 (ObjNum 221 snap 21 seq_num 221) dirty exists 2024-04-20T02:51:50.030 INFO:tasks.rados.rados.0.smithi105.stdout:1245: left oid 50 (ObjNum 221 snap 21 seq_num 221) 2024-04-20T02:51:50.030 INFO:tasks.rados.rados.0.smithi105.stdout:1248: finishing write tid 1 to smithi10548823-42 2024-04-20T02:51:50.030 INFO:tasks.rados.rados.0.smithi105.stdout:1250: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:51:50.072 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:50.072+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:50.166 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:50.166 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:50.173 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:50.174+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:50.176 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:50.205 INFO:tasks.rados.rados.0.smithi105.stdout:1248: finishing write tid 2 to smithi10548823-42 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1248: finishing write tid 3 to smithi10548823-42 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1248: finishing write tid 4 to smithi10548823-42 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1248: finishing write tid 5 to smithi10548823-42 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1248: finishing write tid 6 to smithi10548823-42 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1248: oid 42 updating version 0 to 60 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1248: oid 42 version 60 is already newer than 58 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1248: oid 42 version 60 is already newer than 59 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1248: oid 42 updating version 60 to 61 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1248: oid 42 version 61 is already newer than 57 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 61 (ObjNum 222 snap 21 seq_num 222) dirty exists 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1248: left oid 42 (ObjNum 222 snap 21 seq_num 222) 2024-04-20T02:51:50.206 INFO:tasks.rados.rados.0.smithi105.stdout:1252: expect (ObjNum 186 snap 18 seq_num 186) 2024-04-20T02:51:50.265 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:51:50.265 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:51:50.265 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-20T02:51:50.266 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:51:50.274 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:50.274+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:50.309 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:50.309 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:50.314 INFO:tasks.rados.rados.0.smithi105.stdout:1245: done (6 left) 2024-04-20T02:51:50.314 INFO:tasks.rados.rados.0.smithi105.stdout:1248: done (5 left) 2024-04-20T02:51:50.314 INFO:tasks.rados.rados.0.smithi105.stdout:1249: done (4 left) 2024-04-20T02:51:50.314 INFO:tasks.rados.rados.0.smithi105.stdout:1250: done (3 left) 2024-04-20T02:51:50.314 INFO:tasks.rados.rados.0.smithi105.stdout:1251: done (2 left) 2024-04-20T02:51:50.314 INFO:tasks.rados.rados.0.smithi105.stdout:1252: done (1 left) 2024-04-20T02:51:50.315 INFO:tasks.rados.rados.0.smithi105.stdout:1253: done (0 left) 2024-04-20T02:51:50.315 INFO:tasks.rados.rados.0.smithi105.stdout:1254: write oid 26 current snap is 22 2024-04-20T02:51:50.315 INFO:tasks.rados.rados.0.smithi105.stdout:1254: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-04-20T02:51:50.321 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:50.321 INFO:tasks.rados.rados.0.smithi105.stdout:1254: writing smithi10548823-26 from 547961 to 1102725 tid 1 2024-04-20T02:51:50.327 INFO:tasks.rados.rados.0.smithi105.stdout:1254: writing smithi10548823-26 from 1616308 to 2320048 tid 2 2024-04-20T02:51:50.334 INFO:tasks.rados.rados.0.smithi105.stdout:1254: writing smithi10548823-26 from 2741092 to 3439692 tid 3 2024-04-20T02:51:50.334 INFO:tasks.rados.rados.0.smithi105.stdout:1255: tier_promote oid 12 2024-04-20T02:51:50.334 INFO:tasks.rados.rados.0.smithi105.stdout:1256: read oid 33 snap -1 2024-04-20T02:51:50.334 INFO:tasks.rados.rados.0.smithi105.stdout:1256: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:51:50.334 INFO:tasks.rados.rados.0.smithi105.stdout:1257: set_chunk oid 42 target oid 2024-04-20T02:51:50.375 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:50.375+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:50.379 INFO:tasks.rados.rados.0.smithi105.stdout:1257: set_chunk oid 42 offset: 1402880 length: 1183232 target oid df5e09bf63f131aab0f4c83880f9a07a68528902dce90238f602139c7f3c5a23 offset: 0 snap -1 2024-04-20T02:51:50.380 INFO:tasks.rados.rados.0.smithi105.stdout:1258: read oid 3 snap 21 2024-04-20T02:51:50.380 INFO:tasks.rados.rados.0.smithi105.stdout:1258: expect (ObjNum 211 snap 20 seq_num 211) 2024-04-20T02:51:50.380 INFO:tasks.rados.rados.0.smithi105.stdout:1259: read oid 2 snap -1 2024-04-20T02:51:50.380 INFO:tasks.rados.rados.0.smithi105.stdout:1259: expect (ObjNum 155 snap 10 seq_num 155) 2024-04-20T02:51:50.380 INFO:tasks.rados.rados.0.smithi105.stdout:1260: set_chunk oid 19 target oid 2024-04-20T02:51:50.403 INFO:tasks.rados.rados.0.smithi105.stdout:1260: set_chunk oid 19 offset: 1996800 length: 286208 target oid 13445b01162891fbabb096e2b7099d2951e18b3f9b3f74cec19c2a7b88a7f98a offset: 0 snap -1 2024-04-20T02:51:50.403 INFO:tasks.rados.rados.0.smithi105.stdout:1261: read oid 8 snap 21 2024-04-20T02:51:50.403 INFO:tasks.rados.rados.0.smithi105.stdout:1261: expect (ObjNum 204 snap 19 seq_num 204) 2024-04-20T02:51:50.403 INFO:tasks.rados.rados.0.smithi105.stdout:1262: set_chunk oid 35 target oid 2024-04-20T02:51:50.437 INFO:tasks.rados.rados.0.smithi105.stdout:1262: set_chunk oid 35 offset: 299520 length: 1222144 target oid 5ef94d6566169c46bd6b9981dd4dac0846040cdcf5113ba50143b0b24b137542 offset: 0 snap -1 2024-04-20T02:51:50.437 INFO:tasks.rados.rados.0.smithi105.stdout:1263: read oid 17 snap -1 2024-04-20T02:51:50.437 INFO:tasks.rados.rados.0.smithi105.stdout:1263: expect (ObjNum 216 snap 21 seq_num 216) 2024-04-20T02:51:50.437 INFO:tasks.rados.rados.0.smithi105.stdout:1264: set_chunk oid 32 target oid 2024-04-20T02:51:50.465 INFO:tasks.rados.rados.0.smithi105.stdout:1264: set_chunk oid 32 offset: 137728 length: 1270272 target oid 3198d9e80ae83b5b786423a55ac1bf9867bee3ba114f97d15505dc0881e9a777 offset: 0 snap 21 2024-04-20T02:51:50.465 INFO:tasks.rados.rados.0.smithi105.stdout:1265: read oid 16 snap -1 2024-04-20T02:51:50.465 INFO:tasks.rados.rados.0.smithi105.stdout:1265: expect deleted 2024-04-20T02:51:50.465 INFO:tasks.rados.rados.0.smithi105.stdout:1266: write (excl) oid 23 current snap is 22 2024-04-20T02:51:50.465 INFO:tasks.rados.rados.0.smithi105.stdout:1255: got (0) Success 2024-04-20T02:51:50.465 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 22 (ObjNum 167 snap 14 seq_num 167) dirty exists 2024-04-20T02:51:50.465 INFO:tasks.rados.rados.0.smithi105.stdout:1266: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-04-20T02:51:50.471 INFO:tasks.rados.rados.0.smithi105.stdout:1266: writing smithi10548823-23 from 564768 to 1194781 tid 1 2024-04-20T02:51:50.476 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:50.476+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:50.477 INFO:tasks.rados.rados.0.smithi105.stdout:1266: writing smithi10548823-23 from 1674790 to 2122188 tid 2 2024-04-20T02:51:50.484 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:50.484 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:50.485 INFO:tasks.rados.rados.0.smithi105.stdout:1266: writing smithi10548823-23 from 2768515 to 3394740 tid 3 2024-04-20T02:51:50.485 INFO:tasks.rados.rados.0.smithi105.stdout:1255: done (12 left) 2024-04-20T02:51:50.485 INFO:tasks.rados.rados.0.smithi105.stdout:1267: write (excl) oid 22 current snap is 22 2024-04-20T02:51:50.485 INFO:tasks.rados.rados.0.smithi105.stdout:1267: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-04-20T02:51:50.494 INFO:tasks.rados.rados.0.smithi105.stdout:1267: writing smithi10548823-22 from 581575 to 1286837 tid 1 2024-04-20T02:51:50.498 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:50.500 INFO:tasks.rados.rados.0.smithi105.stdout:1267: writing smithi10548823-22 from 1733272 to 2440681 tid 2 2024-04-20T02:51:50.507 INFO:tasks.rados.rados.0.smithi105.stdout:1267: writing smithi10548823-22 from 3195938 to 3633435 tid 3 2024-04-20T02:51:50.507 INFO:tasks.rados.rados.0.smithi105.stdout:1268: set_chunk oid 9 target oid 2024-04-20T02:51:50.561 INFO:tasks.rados.rados.0.smithi105.stdout:1268: set_chunk oid 9 offset: 675328 length: 1999360 target oid 24a59c40ef607a02467e90e83be767620e52f85e2dd57880354a5e30c0ad193c offset: 0 snap -1 2024-04-20T02:51:50.561 INFO:tasks.rados.rados.0.smithi105.stdout:1269: tier_promote oid 5 2024-04-20T02:51:50.561 INFO:tasks.rados.rados.0.smithi105.stdout:1256: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:51:50.577 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:50.578+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:50.673 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:50.673 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:50.678 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:50.679+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:50.683 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:50.696 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:51:50.697 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:51:50.708 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-04-20T02:51:50.251113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":224018811,"num_objects":154,"num_object_clones":9,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":1784,"num_read_kb":1383874,"num_write":1318,"num_write_kb":348211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":50387536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":29,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":108,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":675696,"kb_used_data":426124,"kb_used_omap":12,"kb_used_meta":245427,"kb_avail":754299024,"statfs":{"total":773094113280,"available":772402200576,"internally_reserved":0,"allocated":436350976,"data_stored":457037338,"data_compressed":10629117,"data_compressed_allocated":11259904,"data_compressed_original":26910720,"omap_allocated":12807,"internal_metadata":251317753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36073596,"num_objects":-21,"num_object_clones":-22,"num_object_copies":-63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":-2,"num_read":50,"num_read_kb":55763,"num_write":50,"num_write_kb":9079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-19,"num_bytes_recovered":-14416424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.022971"},"pg_stats":[{"pgid":"4.1","version":"58'101","reported_seq":379,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.242518+0000","last_change":"2024-04-20T02:51:48.242518+0000","last_active":"2024-04-20T02:51:48.242518+0000","last_peered":"2024-04-20T02:51:48.242518+0000","last_clean":"2024-04-20T02:51:48.242518+0000","last_became_active":"2024-04-20T02:51:18.508043+0000","last_became_peered":"2024-04-20T02:51:18.508043+0000","last_unstale":"2024-04-20T02:51:48.242518+0000","last_undegraded":"2024-04-20T02:51:48.242518+0000","last_fullsized":"2024-04-20T02:51:48.242518+0000","mapping_epoch":46,"log_start":"57'97","ondisk_log_start":"57'97","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'101","last_scrub_stamp":"2024-04-20T02:51:34.098939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:34.098939+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":97,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:56.647060+0000","scrub_duration":0.0019988520000000002,"objects_trimmed":0,"snaptrim_duration":0.00081858500000000004,"stat_sum":{"num_bytes":3479016,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":25085,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-20T02:51:46.231763+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:51:46.231763+0000","last_peered":"2024-04-20T02:51:46.231763+0000","last_clean":"2024-04-20T02:51:46.231763+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:51:46.231763+0000","last_undegraded":"2024-04-20T02:51:46.231763+0000","last_fullsized":"2024-04-20T02:51:46.231763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"88'90","reported_seq":344,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.238673+0000","last_change":"2024-04-20T02:51:47.225357+0000","last_active":"2024-04-20T02:51:47.238673+0000","last_peered":"2024-04-20T02:51:47.238673+0000","last_clean":"2024-04-20T02:51:47.238673+0000","last_became_active":"2024-04-20T02:51:18.505708+0000","last_became_peered":"2024-04-20T02:51:18.505708+0000","last_unstale":"2024-04-20T02:51:47.238673+0000","last_undegraded":"2024-04-20T02:51:47.238673+0000","last_fullsized":"2024-04-20T02:51:47.238673+0000","mapping_epoch":22,"log_start":"86'85","ondisk_log_start":"86'85","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":81,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.922550+0000","scrub_duration":0.001820667,"objects_trimmed":0,"snaptrim_duration":0.99989749000000006,"stat_sum":{"num_bytes":8924484,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":23210,"num_write":13,"num_write_kb":3866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-20T02:51:46.231264+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:51:46.231264+0000","last_peered":"2024-04-20T02:51:46.231264+0000","last_clean":"2024-04-20T02:51:46.231264+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:51:46.231264+0000","last_undegraded":"2024-04-20T02:51:46.231264+0000","last_fullsized":"2024-04-20T02:51:46.231264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.206159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.314643+0000","last_change":"2024-04-20T02:51:41.093997+0000","last_active":"2024-04-20T02:51:47.314643+0000","last_peered":"2024-04-20T02:51:47.314643+0000","last_clean":"2024-04-20T02:51:47.314643+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:51:47.314643+0000","last_undegraded":"2024-04-20T02:51:47.314643+0000","last_fullsized":"2024-04-20T02:51:47.314643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:41.093905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:41.093905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:01.045646+0000","scrub_duration":0.0019242129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"91'206","reported_seq":512,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.498113+0000","last_change":"2024-04-20T02:51:48.248187+0000","last_active":"2024-04-20T02:51:48.498113+0000","last_peered":"2024-04-20T02:51:48.498113+0000","last_clean":"2024-04-20T02:51:48.498113+0000","last_became_active":"2024-04-20T02:51:18.502004+0000","last_became_peered":"2024-04-20T02:51:18.502004+0000","last_unstale":"2024-04-20T02:51:48.498113+0000","last_undegraded":"2024-04-20T02:51:48.498113+0000","last_fullsized":"2024-04-20T02:51:48.498113+0000","mapping_epoch":47,"log_start":"91'203","ondisk_log_start":"91'203","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'157","last_scrub_stamp":"2024-04-20T02:51:19.377186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.377186+0000","objects_scrubbed":5,"log_size":3,"log_dups_size":199,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:38.408177+0000","scrub_duration":0.0026503899999999999,"objects_trimmed":0,"snaptrim_duration":0.0027527430000000002,"stat_sum":{"num_bytes":3680733,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":43,"num_read_kb":69678,"num_write":15,"num_write_kb":4725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"3.1","version":"91'158","reported_seq":260,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.410593+0000","last_change":"2024-04-20T02:50:34.239216+0000","last_active":"2024-04-20T02:51:48.410593+0000","last_peered":"2024-04-20T02:51:48.410593+0000","last_clean":"2024-04-20T02:51:48.410593+0000","last_became_active":"2024-04-20T02:50:34.238906+0000","last_became_peered":"2024-04-20T02:50:34.238906+0000","last_unstale":"2024-04-20T02:51:48.410593+0000","last_undegraded":"2024-04-20T02:51:48.410593+0000","last_fullsized":"2024-04-20T02:51:48.410593+0000","mapping_epoch":21,"log_start":"88'155","ondisk_log_start":"88'155","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":180,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.390119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24111998,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":148,"num_read_kb":6891,"num_write":186,"num_write_kb":56934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.232337+0000","last_change":"2024-04-20T02:51:46.280098+0000","last_active":"2024-04-20T02:51:47.232337+0000","last_peered":"2024-04-20T02:51:47.232337+0000","last_clean":"2024-04-20T02:51:47.232337+0000","last_became_active":"2024-04-20T02:51:46.279750+0000","last_became_peered":"2024-04-20T02:51:46.279750+0000","last_unstale":"2024-04-20T02:51:47.232337+0000","last_undegraded":"2024-04-20T02:51:47.232337+0000","last_fullsized":"2024-04-20T02:51:47.232337+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:28.222203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:28.222203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:54.983534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"88'83","reported_seq":317,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.248213+0000","last_change":"2024-04-20T02:51:48.248213+0000","last_active":"2024-04-20T02:51:48.248213+0000","last_peered":"2024-04-20T02:51:48.248213+0000","last_clean":"2024-04-20T02:51:48.248213+0000","last_became_active":"2024-04-20T02:51:18.504889+0000","last_became_peered":"2024-04-20T02:51:18.504889+0000","last_unstale":"2024-04-20T02:51:48.248213+0000","last_undegraded":"2024-04-20T02:51:48.248213+0000","last_fullsized":"2024-04-20T02:51:48.248213+0000","mapping_epoch":22,"log_start":"85'79","ondisk_log_start":"85'79","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'57","last_scrub_stamp":"2024-04-20T02:51:21.386796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:21.386796+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":78,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:27.553181+0000","scrub_duration":0.0018878359999999999,"objects_trimmed":0,"snaptrim_duration":0.0027467189999999999,"stat_sum":{"num_bytes":3159716,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":11429,"num_write":6,"num_write_kb":1127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"3.2","version":"90'143","reported_seq":335,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.333165+0000","last_change":"2024-04-20T02:51:44.446457+0000","last_active":"2024-04-20T02:51:47.333165+0000","last_peered":"2024-04-20T02:51:47.333165+0000","last_clean":"2024-04-20T02:51:47.333165+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:51:47.333165+0000","last_undegraded":"2024-04-20T02:51:47.333165+0000","last_fullsized":"2024-04-20T02:51:47.333165+0000","mapping_epoch":49,"log_start":"90'138","ondisk_log_start":"90'138","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'130","last_scrub_stamp":"2024-04-20T02:51:44.446332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:51:44.446332+0000","objects_scrubbed":20,"log_size":5,"log_dups_size":146,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:11.809200+0000","scrub_duration":0.0039636890000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20026280,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":129,"num_read_kb":8280,"num_write":167,"num_write_kb":59395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":245,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.314580+0000","last_change":"2024-04-20T02:51:42.076681+0000","last_active":"2024-04-20T02:51:47.314580+0000","last_peered":"2024-04-20T02:51:47.314580+0000","last_clean":"2024-04-20T02:51:47.314580+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:51:47.314580+0000","last_undegraded":"2024-04-20T02:51:47.314580+0000","last_fullsized":"2024-04-20T02:51:47.314580+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:51:42.076506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:51:42.076506+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:59.869381+0000","scrub_duration":0.002077785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.239279+0000","last_change":"2024-04-20T02:51:30.923927+0000","last_active":"2024-04-20T02:51:47.239279+0000","last_peered":"2024-04-20T02:51:47.239279+0000","last_clean":"2024-04-20T02:51:47.239279+0000","last_became_active":"2024-04-20T02:51:01.151544+0000","last_became_peered":"2024-04-20T02:51:01.151544+0000","last_unstale":"2024-04-20T02:51:47.239279+0000","last_undegraded":"2024-04-20T02:51:47.239279+0000","last_fullsized":"2024-04-20T02:51:47.239279+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:30.923833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:30.923833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:47.551592+0000","scrub_duration":0.001311373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"56'33","reported_seq":241,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-20T02:51:46.233251+0000","last_change":"2024-04-20T02:51:46.233251+0000","last_active":"2024-04-20T02:51:46.233251+0000","last_peered":"2024-04-20T02:51:46.233251+0000","last_clean":"2024-04-20T02:51:46.233251+0000","last_became_active":"2024-04-20T02:51:18.502179+0000","last_became_peered":"2024-04-20T02:51:18.502179+0000","last_unstale":"2024-04-20T02:51:46.233251+0000","last_undegraded":"2024-04-20T02:51:46.233251+0000","last_fullsized":"2024-04-20T02:51:46.233251+0000","mapping_epoch":22,"log_start":"53'30","ondisk_log_start":"53'30","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'33","last_scrub_stamp":"2024-04-20T02:51:32.144366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.144366+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:49.222704+0000","scrub_duration":0.01361505,"objects_trimmed":0,"snaptrim_duration":0.00043157900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"}]},{"pgid":"3.0","version":"90'129","reported_seq":215,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.309927+0000","last_change":"2024-04-20T02:50:34.241909+0000","last_active":"2024-04-20T02:51:47.309927+0000","last_peered":"2024-04-20T02:51:47.309927+0000","last_clean":"2024-04-20T02:51:47.309927+0000","last_became_active":"2024-04-20T02:50:34.241613+0000","last_became_peered":"2024-04-20T02:50:34.241613+0000","last_unstale":"2024-04-20T02:51:47.309927+0000","last_undegraded":"2024-04-20T02:51:47.309927+0000","last_fullsized":"2024-04-20T02:51:47.309927+0000","mapping_epoch":21,"log_start":"88'126","ondisk_log_start":"88'126","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":138,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:54.710965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18736650,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":111,"num_read_kb":3312,"num_write":160,"num_write_kb":47207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":90,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-20T02:51:46.276081+0000","last_change":"2024-04-20T02:51:41.175139+0000","last_active":"2024-04-20T02:51:46.276081+0000","last_peered":"2024-04-20T02:51:46.276081+0000","last_clean":"2024-04-20T02:51:46.276081+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:51:46.276081+0000","last_undegraded":"2024-04-20T02:51:46.276081+0000","last_fullsized":"2024-04-20T02:51:46.276081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:41.175035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:41.175035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:46.711136+0000","scrub_duration":0.0014447850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"90'68","reported_seq":306,"reported_epoch":90,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:47.253275+0000","last_change":"2024-04-20T02:51:47.248374+0000","last_active":"2024-04-20T02:51:47.253275+0000","last_peered":"2024-04-20T02:51:47.253275+0000","last_clean":"2024-04-20T02:51:47.253275+0000","last_became_active":"2024-04-20T02:51:47.240817+0000","last_became_peered":"2024-04-20T02:51:47.240817+0000","last_unstale":"2024-04-20T02:51:47.253275+0000","last_undegraded":"2024-04-20T02:51:47.253275+0000","last_fullsized":"2024-04-20T02:51:47.253275+0000","mapping_epoch":46,"log_start":"84'64","ondisk_log_start":"84'64","created":22,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'50","last_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.487770+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":117,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019136573,"objects_trimmed":2,"snaptrim_duration":0.081984028,"stat_sum":{"num_bytes":11787149,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":63,"num_read_kb":84366,"num_write":32,"num_write_kb":9589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2757906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"3.3","version":"91'129","reported_seq":222,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.296437+0000","last_change":"2024-04-20T02:50:34.242061+0000","last_active":"2024-04-20T02:51:48.296437+0000","last_peered":"2024-04-20T02:51:48.296437+0000","last_clean":"2024-04-20T02:51:48.296437+0000","last_became_active":"2024-04-20T02:50:34.241638+0000","last_became_peered":"2024-04-20T02:50:34.241638+0000","last_unstale":"2024-04-20T02:51:48.296437+0000","last_undegraded":"2024-04-20T02:51:48.296437+0000","last_fullsized":"2024-04-20T02:51:48.296437+0000","mapping_epoch":21,"log_start":"90'126","ondisk_log_start":"90'126","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":132,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:58.050023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25339267,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":116,"num_read_kb":4911,"num_write":157,"num_write_kb":45244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.239169+0000","last_change":"2024-04-20T02:51:01.134074+0000","last_active":"2024-04-20T02:51:47.239169+0000","last_peered":"2024-04-20T02:51:47.239169+0000","last_clean":"2024-04-20T02:51:47.239169+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:51:47.239169+0000","last_undegraded":"2024-04-20T02:51:47.239169+0000","last_fullsized":"2024-04-20T02:51:47.239169+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"56'17","reported_seq":215,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-20T02:51:46.232151+0000","last_change":"2024-04-20T02:51:46.232151+0000","last_active":"2024-04-20T02:51:46.232151+0000","last_peered":"2024-04-20T02:51:46.232151+0000","last_clean":"2024-04-20T02:51:46.232151+0000","last_became_active":"2024-04-20T02:51:18.509276+0000","last_became_peered":"2024-04-20T02:51:18.509276+0000","last_unstale":"2024-04-20T02:51:46.232151+0000","last_undegraded":"2024-04-20T02:51:46.232151+0000","last_fullsized":"2024-04-20T02:51:46.232151+0000","mapping_epoch":22,"log_start":"53'14","ondisk_log_start":"53'14","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:51:22.420320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:22.420320+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.257627+0000","scrub_duration":0.0016858400000000001,"objects_trimmed":0,"snaptrim_duration":0.0016430119999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"}]},{"pgid":"4.2","version":"0'0","reported_seq":200,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-20T02:51:46.231672+0000","last_change":"2024-04-20T02:51:46.231672+0000","last_active":"2024-04-20T02:51:46.231672+0000","last_peered":"2024-04-20T02:51:46.231672+0000","last_clean":"2024-04-20T02:51:46.231672+0000","last_became_active":"2024-04-20T02:51:18.506462+0000","last_became_peered":"2024-04-20T02:51:18.506462+0000","last_unstale":"2024-04-20T02:51:46.231672+0000","last_undegraded":"2024-04-20T02:51:46.231672+0000","last_fullsized":"2024-04-20T02:51:46.231672+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:33.440257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:51:33.440257+0000","last_clean_scrub_stamp":"2024-04-20T02:51:33.440257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:00.147584+0000","scrub_duration":0.001747176,"objects_trimmed":0,"snaptrim_duration":0.001193482,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-20T02:51:46.230385+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:51:46.230385+0000","last_peered":"2024-04-20T02:51:46.230385+0000","last_clean":"2024-04-20T02:51:46.230385+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:51:46.230385+0000","last_undegraded":"2024-04-20T02:51:46.230385+0000","last_fullsized":"2024-04-20T02:51:46.230385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:49.728268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"91'71","reported_seq":289,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.343261+0000","last_change":"2024-04-20T02:51:48.274584+0000","last_active":"2024-04-20T02:51:48.343261+0000","last_peered":"2024-04-20T02:51:48.343261+0000","last_clean":"2024-04-20T02:51:48.343261+0000","last_became_active":"2024-04-20T02:51:43.136495+0000","last_became_peered":"2024-04-20T02:51:43.136495+0000","last_unstale":"2024-04-20T02:51:48.343261+0000","last_undegraded":"2024-04-20T02:51:48.343261+0000","last_fullsized":"2024-04-20T02:51:48.343261+0000","mapping_epoch":62,"log_start":"88'67","ondisk_log_start":"88'67","created":22,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'47","last_scrub_stamp":"2024-04-20T02:51:28.917651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:28.917651+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":111,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001846091,"objects_trimmed":0,"snaptrim_duration":0.00043833000000000001,"stat_sum":{"num_bytes":4759965,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":93,"num_read_kb":110607,"num_write":30,"num_write_kb":8039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8110636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.9","version":"91'256","reported_seq":554,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.471024+0000","last_change":"2024-04-20T02:51:48.247276+0000","last_active":"2024-04-20T02:51:48.471024+0000","last_peered":"2024-04-20T02:51:48.471024+0000","last_clean":"2024-04-20T02:51:48.471024+0000","last_became_active":"2024-04-20T02:51:39.132079+0000","last_became_peered":"2024-04-20T02:51:39.132079+0000","last_unstale":"2024-04-20T02:51:48.471024+0000","last_undegraded":"2024-04-20T02:51:48.471024+0000","last_fullsized":"2024-04-20T02:51:48.471024+0000","mapping_epoch":22,"log_start":"91'252","ondisk_log_start":"91'252","created":22,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'238","last_scrub_stamp":"2024-04-20T02:51:45.092632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:45.092632+0000","objects_scrubbed":9,"log_size":4,"log_dups_size":420,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:02.911410+0000","scrub_duration":0.00347503,"objects_trimmed":1,"snaptrim_duration":0.0052990659999999998,"stat_sum":{"num_bytes":16840614,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":26417,"num_write":11,"num_write_kb":3364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.a","version":"90'53","reported_seq":256,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.232442+0000","last_change":"2024-04-20T02:51:46.232507+0000","last_active":"2024-04-20T02:51:47.232442+0000","last_peered":"2024-04-20T02:51:47.232442+0000","last_clean":"2024-04-20T02:51:47.232442+0000","last_became_active":"2024-04-20T02:50:35.413503+0000","last_became_peered":"2024-04-20T02:50:35.413503+0000","last_unstale":"2024-04-20T02:51:47.232442+0000","last_undegraded":"2024-04-20T02:51:47.232442+0000","last_fullsized":"2024-04-20T02:51:47.232442+0000","mapping_epoch":22,"log_start":"84'50","ondisk_log_start":"84'50","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:57.185462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043743100000000001,"stat_sum":{"num_bytes":5176556,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":113,"num_read_kb":127392,"num_write":32,"num_write_kb":8571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"}]},{"pgid":"4.b","version":"88'148","reported_seq":366,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.245992+0000","last_change":"2024-04-20T02:51:48.245992+0000","last_active":"2024-04-20T02:51:48.245992+0000","last_peered":"2024-04-20T02:51:48.245992+0000","last_clean":"2024-04-20T02:51:48.245992+0000","last_became_active":"2024-04-20T02:50:35.242331+0000","last_became_peered":"2024-04-20T02:50:35.242331+0000","last_unstale":"2024-04-20T02:51:48.245992+0000","last_undegraded":"2024-04-20T02:51:48.245992+0000","last_fullsized":"2024-04-20T02:51:48.245992+0000","mapping_epoch":22,"log_start":"86'144","ondisk_log_start":"86'144","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":144,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.773051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065093599999999996,"stat_sum":{"num_bytes":13075813,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":156,"num_read_kb":190816,"num_write":81,"num_write_kb":21776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.c","version":"89'88","reported_seq":269,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.241847+0000","last_change":"2024-04-20T02:51:48.241847+0000","last_active":"2024-04-20T02:51:48.241847+0000","last_peered":"2024-04-20T02:51:48.241847+0000","last_clean":"2024-04-20T02:51:48.241847+0000","last_became_active":"2024-04-20T02:50:35.241480+0000","last_became_peered":"2024-04-20T02:50:35.241480+0000","last_unstale":"2024-04-20T02:51:48.241847+0000","last_undegraded":"2024-04-20T02:51:48.241847+0000","last_fullsized":"2024-04-20T02:51:48.241847+0000","mapping_epoch":22,"log_start":"86'82","ondisk_log_start":"86'82","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":82,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.915698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031446499999999999,"stat_sum":{"num_bytes":3495856,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":52,"num_read_kb":49916,"num_write":47,"num_write_kb":13040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.d","version":"92'155","reported_seq":399,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-20T02:51:49.503545+0000","last_change":"2024-04-20T02:51:48.242556+0000","last_active":"2024-04-20T02:51:49.503545+0000","last_peered":"2024-04-20T02:51:49.503545+0000","last_clean":"2024-04-20T02:51:49.503545+0000","last_became_active":"2024-04-20T02:50:35.528107+0000","last_became_peered":"2024-04-20T02:50:35.528107+0000","last_unstale":"2024-04-20T02:51:49.503545+0000","last_undegraded":"2024-04-20T02:51:49.503545+0000","last_fullsized":"2024-04-20T02:51:49.503545+0000","mapping_epoch":22,"log_start":"92'152","ondisk_log_start":"92'152","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":152,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:59.747989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000678117,"stat_sum":{"num_bytes":11243883,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":247,"num_read_kb":278796,"num_write":84,"num_write_kb":23033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.e","version":"91'80","reported_seq":276,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.512439+0000","last_change":"2024-04-20T02:51:48.242109+0000","last_active":"2024-04-20T02:51:48.512439+0000","last_peered":"2024-04-20T02:51:48.512439+0000","last_clean":"2024-04-20T02:51:48.512439+0000","last_became_active":"2024-04-20T02:50:35.244872+0000","last_became_peered":"2024-04-20T02:50:35.244872+0000","last_unstale":"2024-04-20T02:51:48.512439+0000","last_undegraded":"2024-04-20T02:51:48.512439+0000","last_fullsized":"2024-04-20T02:51:48.512439+0000","mapping_epoch":22,"log_start":"88'77","ondisk_log_start":"88'77","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'77","last_scrub_stamp":"2024-04-20T02:51:46.088382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:46.088382+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":77,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:57.901141+0000","scrub_duration":0.002170693,"objects_trimmed":0,"snaptrim_duration":0.00051101800000000002,"stat_sum":{"num_bytes":5747994,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":46981,"num_write":36,"num_write_kb":11852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.f","version":"88'26","reported_seq":200,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.252195+0000","last_change":"2024-04-20T02:51:47.252439+0000","last_active":"2024-04-20T02:51:47.252195+0000","last_peered":"2024-04-20T02:51:47.252195+0000","last_clean":"2024-04-20T02:51:47.252195+0000","last_became_active":"2024-04-20T02:50:35.245323+0000","last_became_peered":"2024-04-20T02:50:35.245323+0000","last_unstale":"2024-04-20T02:51:47.252195+0000","last_undegraded":"2024-04-20T02:51:47.252195+0000","last_fullsized":"2024-04-20T02:51:47.252195+0000","mapping_epoch":22,"log_start":"82'22","ondisk_log_start":"82'22","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":22,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:53.742736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015133578,"stat_sum":{"num_bytes":3378207,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":39203,"num_write":19,"num_write_kb":5420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"}]},{"pgid":"4.10","version":"90'101","reported_seq":375,"reported_epoch":90,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:47.260850+0000","last_change":"2024-04-20T02:51:47.145273+0000","last_active":"2024-04-20T02:51:47.260850+0000","last_peered":"2024-04-20T02:51:47.260850+0000","last_clean":"2024-04-20T02:51:47.260850+0000","last_became_active":"2024-04-20T02:51:30.132114+0000","last_became_peered":"2024-04-20T02:51:30.132114+0000","last_unstale":"2024-04-20T02:51:47.260850+0000","last_undegraded":"2024-04-20T02:51:47.260850+0000","last_fullsized":"2024-04-20T02:51:47.260850+0000","mapping_epoch":72,"log_start":"88'98","ondisk_log_start":"88'98","created":60,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'78","last_scrub_stamp":"2024-04-20T02:51:32.278907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.278907+0000","objects_scrubbed":5,"log_size":3,"log_dups_size":96,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:41.876076+0000","scrub_duration":0.0025619089999999998,"objects_trimmed":0,"snaptrim_duration":0.91920198099999995,"stat_sum":{"num_bytes":8415100,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":28644,"num_write":7,"num_write_kb":1804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"2"},{"start":"1e","length":"1"}]},{"pgid":"4.11","version":"91'150","reported_seq":508,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.273577+0000","last_change":"2024-04-20T02:51:48.242201+0000","last_active":"2024-04-20T02:51:48.273577+0000","last_peered":"2024-04-20T02:51:48.273577+0000","last_clean":"2024-04-20T02:51:48.273577+0000","last_became_active":"2024-04-20T02:51:32.123982+0000","last_became_peered":"2024-04-20T02:51:32.123982+0000","last_unstale":"2024-04-20T02:51:48.273577+0000","last_undegraded":"2024-04-20T02:51:48.273577+0000","last_fullsized":"2024-04-20T02:51:48.273577+0000","mapping_epoch":74,"log_start":"86'147","ondisk_log_start":"86'147","created":60,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'147","last_scrub_stamp":"2024-04-20T02:51:44.085213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:44.085213+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":147,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:49.057479+0000","scrub_duration":0.0022866549999999999,"objects_trimmed":0,"snaptrim_duration":0.00076309100000000003,"stat_sum":{"num_bytes":9109394,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.12","version":"0'0","reported_seq":218,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-20T02:51:46.230175+0000","last_change":"2024-04-20T02:51:46.230175+0000","last_active":"2024-04-20T02:51:46.230175+0000","last_peered":"2024-04-20T02:51:46.230175+0000","last_clean":"2024-04-20T02:51:46.230175+0000","last_became_active":"2024-04-20T02:51:29.138939+0000","last_became_peered":"2024-04-20T02:51:29.138939+0000","last_unstale":"2024-04-20T02:51:46.230175+0000","last_undegraded":"2024-04-20T02:51:46.230175+0000","last_fullsized":"2024-04-20T02:51:46.230175+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:45.458449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:45.458449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:48.851831+0000","scrub_duration":0.0013346709999999999,"objects_trimmed":0,"snaptrim_duration":0.00055628999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"}]},{"pgid":"4.13","version":"91'226","reported_seq":565,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.365897+0000","last_change":"2024-04-20T02:51:48.245780+0000","last_active":"2024-04-20T02:51:48.365897+0000","last_peered":"2024-04-20T02:51:48.365897+0000","last_clean":"2024-04-20T02:51:48.365897+0000","last_became_active":"2024-04-20T02:51:29.136510+0000","last_became_peered":"2024-04-20T02:51:29.136510+0000","last_unstale":"2024-04-20T02:51:48.365897+0000","last_undegraded":"2024-04-20T02:51:48.365897+0000","last_fullsized":"2024-04-20T02:51:48.365897+0000","mapping_epoch":71,"log_start":"88'223","ondisk_log_start":"88'223","created":60,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'167","last_scrub_stamp":"2024-04-20T02:51:33.169112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:33.169112+0000","objects_scrubbed":10,"log_size":3,"log_dups_size":223,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:39.831181+0000","scrub_duration":0.015460714,"objects_trimmed":0,"snaptrim_duration":0.00057284800000000004,"stat_sum":{"num_bytes":9781674,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":50592,"num_write":6,"num_write_kb":1541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.14","version":"86'28","reported_seq":200,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-20T02:51:46.231404+0000","last_change":"2024-04-20T02:51:46.231404+0000","last_active":"2024-04-20T02:51:46.231404+0000","last_peered":"2024-04-20T02:51:46.231404+0000","last_clean":"2024-04-20T02:51:46.231404+0000","last_became_active":"2024-04-20T02:51:30.144817+0000","last_became_peered":"2024-04-20T02:51:30.144817+0000","last_unstale":"2024-04-20T02:51:46.231404+0000","last_undegraded":"2024-04-20T02:51:46.231404+0000","last_fullsized":"2024-04-20T02:51:46.231404+0000","mapping_epoch":72,"log_start":"78'24","ondisk_log_start":"78'24","created":60,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":24,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00114068,"stat_sum":{"num_bytes":1260542,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":26722,"num_write":10,"num_write_kb":2705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"}]},{"pgid":"4.15","version":"87'56","reported_seq":288,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-20T02:51:47.148494+0000","last_change":"2024-04-20T02:51:47.148494+0000","last_active":"2024-04-20T02:51:47.148494+0000","last_peered":"2024-04-20T02:51:47.148494+0000","last_clean":"2024-04-20T02:51:47.148494+0000","last_became_active":"2024-04-20T02:51:32.124965+0000","last_became_peered":"2024-04-20T02:51:32.124965+0000","last_unstale":"2024-04-20T02:51:47.148494+0000","last_undegraded":"2024-04-20T02:51:47.148494+0000","last_fullsized":"2024-04-20T02:51:47.148494+0000","mapping_epoch":74,"log_start":"84'52","ondisk_log_start":"84'52","created":60,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":52,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.9158328,"stat_sum":{"num_bytes":2150842,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":92387,"num_write":22,"num_write_kb":6348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":16438198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"}]},{"pgid":"4.16","version":"73'67","reported_seq":304,"reported_epoch":91,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:51:48.245428+0000","last_change":"2024-04-20T02:51:30.147512+0000","last_active":"2024-04-20T02:51:48.245428+0000","last_peered":"2024-04-20T02:51:48.245428+0000","last_clean":"2024-04-20T02:51:27.811685+0000","last_became_active":"2024-04-20T02:51:30.142488+0000","last_became_peered":"2024-04-20T02:51:30.142488+0000","last_unstale":"2024-04-20T02:51:48.245428+0000","last_undegraded":"2024-04-20T02:51:48.245428+0000","last_fullsized":"2024-04-20T02:51:48.245428+0000","mapping_epoch":72,"log_start":"73'64","ondisk_log_start":"73'64","created":60,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099853799999999999,"stat_sum":{"num_bytes":9746710,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":73,"num_read_kb":78139,"num_write":38,"num_write_kb":10215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8853601,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":23,"stat_sum":{"num_bytes":135214248,"num_objects":47,"num_object_clones":9,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":1164,"num_read_kb":1360380,"num_write":489,"num_write_kb":137015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":36160341,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":29,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":281198592,"data_stored":290856761,"data_compressed":1160982,"data_compressed_allocated":995328,"data_compressed_original":2285568,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":69,"acting":69,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":88214195,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":504,"num_read_kb":23394,"num_write":670,"num_write_kb":208780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":145580032,"data_stored":158153881,"data_compressed":9437440,"data_compressed_allocated":10162176,"data_compressed_original":23191552,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509464,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75992,"kb_used_data":42356,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94295848,"statfs":{"total":96636764160,"available":96558948352,"internally_reserved":0,"allocated":43372544,"data_stored":44377691,"data_compressed":295417,"data_compressed_allocated":294912,"data_compressed_original":1064960,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.378},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.42199999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.29999999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.36499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.436},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.35799999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757260,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70396,"kb_used_data":38572,"kb_used_omap":1,"kb_used_meta":31806,"kb_avail":94301444,"statfs":{"total":96636764160,"available":96564678656,"internally_reserved":0,"allocated":39497728,"data_stored":41808503,"data_compressed":761665,"data_compressed_allocated":815104,"data_compressed_original":1708032,"omap_allocated":1657,"internal_metadata":32569735},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49584,"kb_used_data":22556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94322256,"statfs":{"total":96636764160,"available":96585990144,"internally_reserved":0,"allocated":23097344,"data_stored":23441654,"data_compressed":829349,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.25900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.32400000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.33600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":11.877000000000001,"5min":11.877000000000001,"15min":11.877000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.36699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.378},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":11.805,"5min":11.805,"15min":11.805},"last":0.41699999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.443}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88536,"kb_used_data":57412,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94283304,"statfs":{"total":96636764160,"available":96546103296,"internally_reserved":0,"allocated":58789888,"data_stored":61750572,"data_compressed":836910,"data_compressed_allocated":905216,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.42399999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.378},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.439}]},{"osd":4,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.753},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.57199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.69299999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.77900000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91612,"kb_used_data":60496,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94280228,"statfs":{"total":96636764160,"available":96542953472,"internally_reserved":0,"allocated":61947904,"data_stored":64858501,"data_compressed":1797097,"data_compressed_allocated":1875968,"data_compressed_original":4009984,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.23400000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.193},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.247}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.26600000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.36299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.29799999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.40500000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90684,"kb_used_data":59568,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94281156,"statfs":{"total":96636764160,"available":96543903744,"internally_reserved":0,"allocated":60997632,"data_stored":63765309,"data_compressed":2403275,"data_compressed_allocated":2613248,"data_compressed_original":6434816,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":2.2709999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.0369999999999999,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":1.756}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.88500000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":9.9570000000000007,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":1.1279999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.8679999999999999,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.98,"5min":3.98,"15min":3.98},"last":0.29799999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509460,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105156,"kb_used_data":72548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94266684,"statfs":{"total":96636764160,"available":96529084416,"internally_reserved":0,"allocated":74289152,"data_stored":81532703,"data_compressed":1991668,"data_compressed_allocated":2015232,"data_compressed_original":5332992,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.378}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.317},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.29999999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.33600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.39300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.47399999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.43099999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509462,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103736,"kb_used_data":72616,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94268104,"statfs":{"total":96636764160,"available":96530538496,"internally_reserved":0,"allocated":74358784,"data_stored":75502405,"data_compressed":1713736,"data_compressed_allocated":1851392,"data_compressed_original":4296704,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.372}]},{"osd":1,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.32100000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.35999999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.438}]},{"osd":7,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27889664,"data_stored":30158031,"data_compressed":1629136,"data_compressed_allocated":1757184,"data_compressed_original":4100096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10592256,"data_stored":11644619,"data_compressed":752310,"data_compressed_allocated":811008,"data_compressed_original":1900544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10592256,"data_stored":11644619,"data_compressed":752310,"data_compressed_allocated":811008,"data_compressed_original":1900544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22446080,"data_stored":24219568,"data_compressed":1506998,"data_compressed_allocated":1613824,"data_compressed_original":3457024,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":27889664,"data_stored":30158031,"data_compressed":1629136,"data_compressed_allocated":1757184,"data_compressed_original":4100096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10231808,"data_stored":11081881,"data_compressed":761665,"data_compressed_allocated":815104,"data_compressed_original":1708032,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":35938304,"data_stored":39247132,"data_compressed":2405885,"data_compressed_allocated":2596864,"data_compressed_original":6025216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42725376,"data_stored":46577625,"data_compressed":357214,"data_compressed_allocated":225280,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47165440,"data_stored":49660165,"data_compressed":84600,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11100160,"data_stored":10991130,"data_compressed":54370,"data_compressed_allocated":57344,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38469632,"data_stored":40193145,"data_compressed":290099,"data_compressed_allocated":262144,"data_compressed_original":552960,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45973504,"data_stored":45438333,"data_compressed":84600,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29151232,"data_stored":31199316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25509888,"data_stored":25138800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":41103360,"data_stored":41658247,"data_compressed":290099,"data_compressed_allocated":262144,"data_compressed_original":552960,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:51:50.711 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-20T02:51:50.711 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-04-20T02:51:50.728 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:50.746 INFO:tasks.rados.rados.0.smithi105.stdout:1254: finishing write tid 1 to smithi10548823-26 2024-04-20T02:51:50.747 INFO:tasks.rados.rados.0.smithi105.stdout:1256: done (14 left) 2024-04-20T02:51:50.747 INFO:tasks.rados.rados.0.smithi105.stdout:1270: write oid 30 current snap is 22 2024-04-20T02:51:50.747 INFO:tasks.rados.rados.0.smithi105.stdout:1257: finishing set_chunk to oid 42 2024-04-20T02:51:50.747 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 61 (ObjNum 222 snap 21 seq_num 222) dirty exists 2024-04-20T02:51:50.747 INFO:tasks.rados.rados.0.smithi105.stdout:1259: expect (ObjNum 155 snap 10 seq_num 155) 2024-04-20T02:51:50.791 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:50.791+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:50.829 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:50.829 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:50.841 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:50.881 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:50.881+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:50.886 INFO:tasks.rados.rados.0.smithi105.stdout:1258: expect (ObjNum 211 snap 20 seq_num 211) 2024-04-20T02:51:50.900 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:50.920 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:50.939 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:50.958 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:50.979 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:50.981 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:50.982+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:50.999 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:51.014 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:51.014 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:51.014 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:51.014 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:51.014 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:51.024 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:51.026 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:51.039 INFO:tasks.rados.rados.0.smithi105.stdout:1254: finishing write tid 2 to smithi10548823-26 2024-04-20T02:51:51.039 INFO:tasks.rados.rados.0.smithi105.stdout:1254: finishing write tid 3 to smithi10548823-26 2024-04-20T02:51:51.039 INFO:tasks.rados.rados.0.smithi105.stdout:1254: finishing write tid 4 to smithi10548823-26 2024-04-20T02:51:51.039 INFO:tasks.rados.rados.0.smithi105.stdout:1254: finishing write tid 5 to smithi10548823-26 2024-04-20T02:51:51.039 INFO:tasks.rados.rados.0.smithi105.stdout:1254: oid 26 updating version 0 to 261 2024-04-20T02:51:51.039 INFO:tasks.rados.rados.0.smithi105.stdout:1254: oid 26 version 261 is already newer than 260 2024-04-20T02:51:51.039 INFO:tasks.rados.rados.0.smithi105.stdout:1254: oid 26 version 261 is already newer than 259 2024-04-20T02:51:51.039 INFO:tasks.rados.rados.0.smithi105.stdout:1254: oid 26 version 261 is already newer than 258 2024-04-20T02:51:51.039 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 261 (ObjNum 223 snap 22 seq_num 223) dirty exists 2024-04-20T02:51:51.040 INFO:tasks.rados.rados.0.smithi105.stdout:1254: left oid 26 (ObjNum 223 snap 22 seq_num 223) 2024-04-20T02:51:51.040 INFO:tasks.rados.rados.0.smithi105.stdout:1260: finishing set_chunk to oid 19 2024-04-20T02:51:51.040 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 47 (ObjNum 88 snap 2 seq_num 88) dirty exists 2024-04-20T02:51:51.040 INFO:tasks.rados.rados.0.smithi105.stdout:1261: expect (ObjNum 204 snap 19 seq_num 204) 2024-04-20T02:51:51.049 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:51.105 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:51.105+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:51.163 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:51.163 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:51.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:51.163 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.577514970999999, 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:51.164 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:51:51.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:51:51.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.170 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.561496692999995, 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:51:51.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:51.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:51:51.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:51.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.176 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.177 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.177 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:51.177 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:51.177 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.561343464000004, 2024-04-20T02:51:51.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:51:51.177 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.177 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:51:51.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:51:51.183 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.561233455000007, 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.184 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:51:51.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:51.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.189 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.561203078999995, 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.190 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:51:51.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:51:51.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.556785660000003, 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:51:51.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.200 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.547533908999995, 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.201 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:51:51.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:51:51.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:51:51.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:51.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:51:51.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.545011766000002, 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.208 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:51.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:51:51.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:51:51.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:51:51.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:51:51.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.544926899000004, 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:51:51.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:51:51.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.544909289000003, 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:51:51.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:991 4.b e32/22 4:d4308fef:::smithi10548823-6:head v 32'29, mlcod=30'27)", 2024-04-20T02:51:51.227 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.438005035000003, 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046400134000000003, 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724650+0000", 2024-04-20T02:51:51.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724649+0000", 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724907+0000", 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025779699999999998 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724924+0000", 2024-04-20T02:51:51.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7110000000000001e-05 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724934+0000", 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5370000000000003e-06 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.724984+0000", 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0791000000000002e-05 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.725011+0000", 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6299e-05 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771009+0000", 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045998144999999997 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.771050+0000", 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1727999999999998e-05 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.195542680999999, 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058137664999999998, 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967112+0000", 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981895+0000", 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014783522 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981906+0000", 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0560000000000001e-05 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981912+0000", 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981956+0000", 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3455000000000003e-05 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981979+0000", 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3475000000000001e-05 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025222+0000", 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043242622000000001 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025250+0000", 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8520000000000001e-05 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.235 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.750534692999999, 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:51:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.744529905, 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.240 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:51:51.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:51:51.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:130 3.0 3:1b476ba0:::474a8d6c2b8981e0c13c883e0d254815cfd155e523c9b654ff4e5007d0a1fc09:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.035188898000001, 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045176642000000003, 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127466+0000", 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128171+0000", 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070576600000000003 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128183+0000", 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1542999999999999e-05 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128191+0000", 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5839999999999995e-06 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134592+0000", 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006400233 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134784+0000", 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000191831 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135041+0000", 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000257123 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162360+0000", 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027319774000000002 2024-04-20T02:51:51.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162476+0000", 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115505 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172550+0000", 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01007366 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172598+0000", 2024-04-20T02:51:51.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8216000000000001e-05 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.172643+0000", 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5305000000000001e-05 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:135 3.0 3:12659911:::ad802ca609ac9a6b9a438bdacacdb4fb27f17cd228f3ff5506eb31f643d749ff:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.030160942000002, 2024-04-20T02:51:51.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045928208999999998, 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132494+0000", 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.132493+0000", 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133291+0000", 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079715799999999996 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133301+0000", 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0533000000000001e-05 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133309+0000", 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9910000000000002e-06 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151891+0000", 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018582155 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152003+0000", 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112064 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152211+0000", 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020773100000000001 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.162492+0000", 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010281047999999999 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169484+0000", 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006991929 2024-04-20T02:51:51.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178312+0000", 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0088282789999999996 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178374+0000", 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2123000000000001e-05 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178423+0000", 2024-04-20T02:51:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201999999999997e-05 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:138 3.0 3:2a051c65:::f4051f504bf0c69c0b0fcc37af36dc6836e32eef2351291df998149766aaa9b9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.029339524999997, 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045257957000000001, 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133316+0000", 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133315+0000", 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133606+0000", 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029134199999999999 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133615+0000", 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2479999999999996e-06 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.133621+0000", 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7739999999999999e-06 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159641+0000", 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026020008000000001 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159813+0000", 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017143200000000001 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 2,6", 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160027+0000", 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021374400000000001 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.163217+0000", 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0031905869999999999 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169517+0000", 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0063002290000000001 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178484+0000", 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0089668070000000002 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178534+0000", 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9654000000000003e-05 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.178574+0000", 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9929999999999999e-05 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:51.261 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.074021774999999, 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049605908999999997, 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088634+0000", 2024-04-20T02:51:51.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088632+0000", 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090096+0000", 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001463675 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090112+0000", 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5557999999999999e-05 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090125+0000", 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3794999999999999e-05 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090196+0000", 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0537000000000006e-05 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090224+0000", 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8555999999999999e-05 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138207+0000", 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047982536999999999 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.138240+0000", 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2480000000000001e-05 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.265 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.021313782, 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056137746000000002, 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141340+0000", 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142751+0000", 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014113540000000001 2024-04-20T02:51:51.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142770+0000", 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8450000000000001e-05 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142780+0000", 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0284e-05 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142982+0000", 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000202244 2024-04-20T02:51:51.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.143007+0000", 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4929000000000001e-05 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197446+0000", 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054438519999999997 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197479+0000", 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3597999999999999e-05 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2490 4.9 e90/81 4:9c87d36d:::smithi10548823-24:head v 90'246, mlcod=88'242)", 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.9182148589999999, 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055338261999999999, 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.270 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244440+0000", 2024-04-20T02:51:51.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245684+0000", 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001243755 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245696+0000", 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.171e-05 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245706+0000", 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0236e-05 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245756+0000", 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0571000000000001e-05 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245784+0000", 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7841999999999999e-05 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.299737+0000", 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053952420000000001 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.299779+0000", 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2014000000000001e-05 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.274 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:51.275 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:51:51.277 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:51.277 INFO:tasks.rados.rados.0.smithi105.stdout:1262: finishing set_chunk to oid 35 2024-04-20T02:51:51.277 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 299520 length: 1222144 2024-04-20T02:51:51.277 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 124 snap 9 seq_num 124) dirty exists 2024-04-20T02:51:51.277 INFO:tasks.rados.rados.0.smithi105.stdout:1263: expect (ObjNum 216 snap 21 seq_num 216) 2024-04-20T02:51:51.278 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:51.276+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:51.377 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:51.378+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:51.450 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:51.450 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:51.450 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:51.450 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:51.454 INFO:tasks.rados.rados.0.smithi105.stdout:1264: finishing set_chunk to oid 32 2024-04-20T02:51:51.454 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 137728 length: 1270272 2024-04-20T02:51:51.454 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 183 snap 17 seq_num 183) dirty exists 2024-04-20T02:51:51.454 INFO:tasks.rados.rados.0.smithi105.stdout:1266: finishing write tid 1 to smithi10548823-23 2024-04-20T02:51:51.454 INFO:tasks.rados.rados.0.smithi105.stdout:1266: finishing write tid 2 to smithi10548823-23 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1266: finishing write tid 3 to smithi10548823-23 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1266: finishing write tid 4 to smithi10548823-23 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1266: finishing write tid 5 to smithi10548823-23 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1266: oid 23 updating version 0 to 86 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1266: oid 23 version 86 is already newer than 85 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1266: oid 23 updating version 86 to 88 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1266: oid 23 version 88 is already newer than 87 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 88 (ObjNum 224 snap 22 seq_num 224) dirty exists 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1266: left oid 23 (ObjNum 224 snap 22 seq_num 224) 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1269: got (0) Success 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 116 (ObjNum 186 snap 18 seq_num 186) dirty exists 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1268: finishing set_chunk to oid 9 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 161 (ObjNum 177 snap 14 seq_num 177) dirty exists 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1267: finishing write tid 1 to smithi10548823-22 2024-04-20T02:51:51.455 INFO:tasks.rados.rados.0.smithi105.stdout:1270: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-04-20T02:51:51.461 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:51.462 INFO:tasks.rados.rados.0.smithi105.stdout:1270: writing smithi10548823-30 from 598382 to 1378893 tid 1 2024-04-20T02:51:51.469 INFO:tasks.rados.rados.0.smithi105.stdout:1270: writing smithi10548823-30 from 2075401 to 2526468 tid 2 2024-04-20T02:51:51.477 INFO:tasks.rados.rados.0.smithi105.stdout:1270: writing smithi10548823-30 from 3107008 to 3755777 tid 3 2024-04-20T02:51:51.477 INFO:tasks.rados.rados.0.smithi105.stdout:1254: done (14 left) 2024-04-20T02:51:51.477 INFO:tasks.rados.rados.0.smithi105.stdout:1257: done (13 left) 2024-04-20T02:51:51.477 INFO:tasks.rados.rados.0.smithi105.stdout:1258: done (12 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1259: done (11 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1260: done (10 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1261: done (9 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1262: done (8 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1263: done (7 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1264: done (6 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1265: done (5 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1266: done (4 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1268: done (3 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1269: done (2 left) 2024-04-20T02:51:51.478 INFO:tasks.rados.rados.0.smithi105.stdout:1271: set_chunk oid 46 target oid 2024-04-20T02:51:51.479 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:51.478+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:51.551 INFO:tasks.rados.rados.0.smithi105.stdout:1271: set_chunk oid 46 offset: 561152 length: 397312 target oid 09c9bb3e2b154d63c31c39d19ca427e5180dba7abb070768e229e2cbe05caef3 offset: 0 snap -1 2024-04-20T02:51:51.551 INFO:tasks.rados.rados.0.smithi105.stdout:1272: read oid 8 snap -1 2024-04-20T02:51:51.551 INFO:tasks.rados.rados.0.smithi105.stdout:1272: expect (ObjNum 204 snap 19 seq_num 204) 2024-04-20T02:51:51.551 INFO:tasks.rados.rados.0.smithi105.stdout:1273: read oid 48 snap -1 2024-04-20T02:51:51.551 INFO:tasks.rados.rados.0.smithi105.stdout:1273: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:51:51.551 INFO:tasks.rados.rados.0.smithi105.stdout:1274: write oid 21 current snap is 22 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1267: finishing write tid 2 to smithi10548823-22 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1267: finishing write tid 3 to smithi10548823-22 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1267: finishing write tid 4 to smithi10548823-22 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1267: finishing write tid 5 to smithi10548823-22 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1267: oid 22 updating version 0 to 72 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1267: oid 22 version 72 is already newer than 70 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1267: oid 22 version 72 is already newer than 71 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1267: oid 22 version 72 is already newer than 69 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 72 (ObjNum 225 snap 22 seq_num 225) dirty exists 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1267: left oid 22 (ObjNum 225 snap 22 seq_num 225) 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1270: finishing write tid 1 to smithi10548823-30 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1270: finishing write tid 2 to smithi10548823-30 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1270: finishing write tid 3 to smithi10548823-30 2024-04-20T02:51:51.552 INFO:tasks.rados.rados.0.smithi105.stdout:1270: finishing write tid 4 to smithi10548823-30 2024-04-20T02:51:51.553 INFO:tasks.rados.rados.0.smithi105.stdout:1270: finishing write tid 5 to smithi10548823-30 2024-04-20T02:51:51.553 INFO:tasks.rados.rados.0.smithi105.stdout:1270: oid 30 updating version 0 to 75 2024-04-20T02:51:51.553 INFO:tasks.rados.rados.0.smithi105.stdout:1270: oid 30 version 75 is already newer than 74 2024-04-20T02:51:51.553 INFO:tasks.rados.rados.0.smithi105.stdout:1270: oid 30 updating version 75 to 76 2024-04-20T02:51:51.553 INFO:tasks.rados.rados.0.smithi105.stdout:1270: oid 30 updating version 76 to 77 2024-04-20T02:51:51.553 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 77 (ObjNum 226 snap 22 seq_num 226) dirty exists 2024-04-20T02:51:51.553 INFO:tasks.rados.rados.0.smithi105.stdout:1270: left oid 30 (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:51:51.553 INFO:tasks.rados.rados.0.smithi105.stdout:1274: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-04-20T02:51:51.558 INFO:tasks.rados.rados.0.smithi105.stdout:1274: writing smithi10548823-21 from 615189 to 1070949 tid 1 2024-04-20T02:51:51.567 INFO:tasks.rados.rados.0.smithi105.stdout:1274: writing smithi10548823-21 from 1733883 to 2444961 tid 2 2024-04-20T02:51:51.574 INFO:tasks.rados.rados.0.smithi105.stdout:1274: writing smithi10548823-21 from 3134431 to 3594472 tid 3 2024-04-20T02:51:51.574 INFO:tasks.rados.rados.0.smithi105.stdout:1267: done (5 left) 2024-04-20T02:51:51.575 INFO:tasks.rados.rados.0.smithi105.stdout:1270: done (4 left) 2024-04-20T02:51:51.575 INFO:tasks.rados.rados.0.smithi105.stdout:1275: tier_evict oid 47 2024-04-20T02:51:51.575 INFO:tasks.rados.rados.0.smithi105.stdout:1275: tier_evict oid 47 snap -1 2024-04-20T02:51:51.575 INFO:tasks.rados.rados.0.smithi105.stdout:1276: set_chunk oid 39 target oid 2024-04-20T02:51:51.578 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:51.579+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:51.615 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:51.615 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:51.615 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:51.615 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:51.615 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:51.625 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:51.641 INFO:tasks.rados.rados.0.smithi105.stdout:1276: set_chunk oid 39 offset: 172032 length: 1502208 target oid c1f8b2145aa72e4029203cbe2b2d99b68764ae8c33b1e68b5f958a826e079bb6 offset: 0 snap 21 2024-04-20T02:51:51.641 INFO:tasks.rados.rados.0.smithi105.stdout:1277: tier_evict oid 44 2024-04-20T02:51:51.641 INFO:tasks.rados.rados.0.smithi105.stdout:1277: tier_evict oid 44 snap -1 2024-04-20T02:51:51.642 INFO:tasks.rados.rados.0.smithi105.stdout:1271: finishing set_chunk to oid 46 2024-04-20T02:51:51.642 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 48 (ObjNum 192 snap 19 seq_num 192) dirty exists 2024-04-20T02:51:51.642 INFO:tasks.rados.rados.0.smithi105.stdout:1273: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:51:51.679 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:51.679+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.826288972, 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041399200999999997, 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.772 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945227+0000", 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947429+0000", 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022028989999999999 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947448+0000", 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.821e-05 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947460+0000", 2024-04-20T02:51:51.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2373e-05 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947511+0000", 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0844999999999999e-05 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947531+0000", 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0017000000000002e-05 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986580+0000", 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039049213999999999 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986627+0000", 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7070000000000002e-05 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1145 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'30, mlcod=34'26)", 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.807558272999998, 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026752594000000001, 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:51.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:51.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963958+0000", 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964662+0000", 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070467299999999995 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964678+0000", 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5597999999999999e-05 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964687+0000", 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2979999999999997e-06 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968184+0000", 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034972250000000001 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968211+0000", 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6829999999999999e-05 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990640+0000", 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022429317000000001 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990711+0000", 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0794e-05 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:129 3.0 e42/21 3:0256f537:::3f82a762bf2204de901e10565f70e759ee2a26bf55ffef4c8d5027a47e96ec31:head v 42'78, mlcod=42'76)", 2024-04-20T02:51:51.781 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.636869697999998, 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024418002000000001, 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134646+0000", 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012265099999999999 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134789+0000", 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9924000000000001e-05 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134799+0000", 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0042e-05 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134842+0000", 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3791000000000001e-05 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134866+0000", 2024-04-20T02:51:51.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3791e-05 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159017+0000", 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024150919 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159065+0000", 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201000000000001e-05 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.786 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:134 3.0 e42/21 3:33aaac3f:::d0aca4dab23a9f22fb58211e350c7c1c1b7601bafbb496b93afbbc91e2ed921c:head v 42'81, mlcod=42'76)", 2024-04-20T02:51:51.786 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:51.786 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.626847499, 2024-04-20T02:51:51.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024591152000000002, 2024-04-20T02:51:51.786 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.786 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.786 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144949+0000", 2024-04-20T02:51:51.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028058500000000002 2024-04-20T02:51:51.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144959+0000", 2024-04-20T02:51:51.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0335e-05 2024-04-20T02:51:51.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144967+0000", 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6040000000000004e-06 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151637+0000", 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066694670000000001 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151659+0000", 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2855000000000001e-05 2024-04-20T02:51:51.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169224+0000", 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017565041 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169261+0000", 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6179e-05 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1462 3.0 e44/21 3:29b349f1:::ea1112aea5e7be379214f0429a6e898cf4f4a05bdbb29b6cd15764b4776418b1:head v 44'85, mlcod=42'83)", 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.593727614000002, 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034060275000000001, 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178932+0000", 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011426069999999999 2024-04-20T02:51:51.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178944+0000", 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.287e-05 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178962+0000", 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7669999999999999e-05 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179013+0000", 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1295000000000003e-05 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179038+0000", 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4859999999999999e-05 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211813+0000", 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032775190000000003 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211850+0000", 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6216e-05 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1629 3.0 e51/21 3:088cc2f3:::a1744cc895df5e9d3e624b281fcb9edc57f3f342d1f108d4c38dc778f219a41a:head v 51'93, mlcod=45'91)", 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.968428277000001, 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035323188999999998, 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.795 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803088+0000", 2024-04-20T02:51:51.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804187+0000", 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00109891 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804201+0000", 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3772e-05 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804213+0000", 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2335e-05 2024-04-20T02:51:51.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804285+0000", 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1271000000000003e-05 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804306+0000", 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1359999999999999e-05 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838377+0000", 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034070994 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838412+0000", 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4790999999999999e-05 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1653 4.6 e51/22 4:697da67d:::smithi10548823-48:head v 51'41, mlcod=45'37)", 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.346689169000001, 2024-04-20T02:51:51.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037653640000000002, 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424827+0000", 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425066+0000", 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023957399999999999 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425117+0000", 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1082999999999999e-05 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425146+0000", 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8129000000000001e-05 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425217+0000", 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1402000000000005e-05 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425278+0000", 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1498e-05 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462456+0000", 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037177106000000001 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462481+0000", 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5851e-05 2024-04-20T02:51:51.804 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1786 4.d e56/22 4:b376c20e:::smithi10548823-49:head v 56'80, mlcod=55'77)", 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.959597631999998, 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028136330000000001, 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:51.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:51.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812644+0000", 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072510100000000002 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812656+0000", 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1688999999999999e-05 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812663+0000", 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812707+0000", 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4573999999999997e-05 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812722+0000", 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5319e-05 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840021+0000", 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027298111999999999 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840056+0000", 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5086000000000003e-05 2024-04-20T02:51:51.809 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1799 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'53, mlcod=54'49)", 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.697268610999998, 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050527214000000001, 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.810 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:51.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074614+0000", 2024-04-20T02:51:51.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036631499999999998 2024-04-20T02:51:51.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074624+0000", 2024-04-20T02:51:51.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8330000000000006e-06 2024-04-20T02:51:51.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074632+0000", 2024-04-20T02:51:51.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3599999999999998e-06 2024-04-20T02:51:51.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074683+0000", 2024-04-20T02:51:51.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1137999999999997e-05 2024-04-20T02:51:51.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074704+0000", 2024-04-20T02:51:51.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1616000000000001e-05 2024-04-20T02:51:51.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124714+0000", 2024-04-20T02:51:51.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050009642 2024-04-20T02:51:51.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124776+0000", 2024-04-20T02:51:51.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1581000000000006e-05 2024-04-20T02:51:51.816 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.816 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.816 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.817 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:51.817 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:51.817 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.682772515000003, 2024-04-20T02:51:51.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046184680999999998, 2024-04-20T02:51:51.818 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.818 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.818 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:51.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:51.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088743+0000", 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090327+0000", 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001584071 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090343+0000", 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5193e-05 2024-04-20T02:51:51.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090354+0000", 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1336e-05 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090411+0000", 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6589999999999999e-05 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090431+0000", 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0579000000000001e-05 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134883+0000", 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044451877000000001 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134929+0000", 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6131000000000003e-05 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1800 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'54, mlcod=54'49)", 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.670366440999999, 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027029681999999999, 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101150+0000", 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101284+0000", 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000134688 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101295+0000", 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101305+0000", 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0930999999999999e-05 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101347+0000", 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1522e-05 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101369+0000", 2024-04-20T02:51:51.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2027000000000001e-05 2024-04-20T02:51:51.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128138+0000", 2024-04-20T02:51:51.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026768626 2024-04-20T02:51:51.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128180+0000", 2024-04-20T02:51:51.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2568999999999998e-05 2024-04-20T02:51:51.832 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.832 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.832 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.833 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1807 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'87, mlcod=56'82)", 2024-04-20T02:51:51.833 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:51.833 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.662685615000001, 2024-04-20T02:51:51.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026347280000000001, 2024-04-20T02:51:51.833 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.833 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.833 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.834 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.834 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:51.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:51.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108830+0000", 2024-04-20T02:51:51.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:51.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109221+0000", 2024-04-20T02:51:51.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039087699999999998 2024-04-20T02:51:51.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.837 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109239+0000", 2024-04-20T02:51:51.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8904000000000001e-05 2024-04-20T02:51:51.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.838 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.838 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109271+0000", 2024-04-20T02:51:51.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1319000000000003e-05 2024-04-20T02:51:51.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120423+0000", 2024-04-20T02:51:51.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011151912999999999 2024-04-20T02:51:51.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120454+0000", 2024-04-20T02:51:51.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1464999999999998e-05 2024-04-20T02:51:51.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135158+0000", 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014703554000000001 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135179+0000", 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0965999999999999e-05 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2038 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'159, mlcod=0'0)", 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.189897041999998, 2024-04-20T02:51:51.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044306238999999997, 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581964+0000", 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034420099999999998 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581980+0000", 2024-04-20T02:51:51.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6578999999999998e-05 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581991+0000", 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0949e-05 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582734+0000", 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074277299999999996 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582761+0000", 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6627000000000001e-05 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625882+0000", 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043121617000000001 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625926+0000", 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3838999999999998e-05 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.846 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2039 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'160, mlcod=0'0)", 2024-04-20T02:51:51.846 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:51.846 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.169498956999998, 2024-04-20T02:51:51.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027318719000000002, 2024-04-20T02:51:51.846 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.846 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.846 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:51.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:51.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602017+0000", 2024-04-20T02:51:51.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:51.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.849 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.849 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602364+0000", 2024-04-20T02:51:51.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000347139 2024-04-20T02:51:51.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.849 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.849 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602375+0000", 2024-04-20T02:51:51.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.128e-05 2024-04-20T02:51:51.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602384+0000", 2024-04-20T02:51:51.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2600000000000005e-06 2024-04-20T02:51:51.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605223+0000", 2024-04-20T02:51:51.851 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0028393400000000001 2024-04-20T02:51:51.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.851 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605245+0000", 2024-04-20T02:51:51.851 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2078000000000001e-05 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629297+0000", 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024051605 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629337+0000", 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0126000000000002e-05 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.852 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2137 3.0 e70/21 3:0d26903d:::7146376437549e0b99264d159bf1e9c8bc43ac1c7eb519f65899cfd592a43a4a:head v 70'98, mlcod=55'96)", 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.987370973000001, 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024394852000000002, 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784144+0000", 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785280+0000", 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001135741 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785299+0000", 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8845999999999999e-05 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785310+0000", 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1450999999999999e-05 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785362+0000", 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1798000000000001e-05 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785388+0000", 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5763999999999999e-05 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808507+0000", 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023118742000000001 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808541+0000", 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4258000000000003e-05 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2142 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'65, mlcod=66'61)", 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.954671560000001, 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033365672999999998, 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:51.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816981+0000", 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135952 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816994+0000", 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3405e-05 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817003+0000", 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4319999999999999e-06 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817055+0000", 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1842999999999998e-05 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817076+0000", 2024-04-20T02:51:51.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1375000000000001e-05 2024-04-20T02:51:51.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.861 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.861 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850179+0000", 2024-04-20T02:51:51.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033103051000000001 2024-04-20T02:51:51.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.862 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850211+0000", 2024-04-20T02:51:51.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1896000000000001e-05 2024-04-20T02:51:51.862 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.862 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.862 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.863 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2143 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'66, mlcod=66'61)", 2024-04-20T02:51:51.863 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:51.863 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.940805718, 2024-04-20T02:51:51.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023949208999999999, 2024-04-20T02:51:51.863 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.863 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.863 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:51.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830711+0000", 2024-04-20T02:51:51.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830710+0000", 2024-04-20T02:51:51.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830883+0000", 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017331000000000001 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830903+0000", 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0043000000000001e-05 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.830914+0000", 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1263999999999999e-05 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.836558+0000", 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056437309999999999 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.836587+0000", 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.854633+0000", 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01804584 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.854660+0000", 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7708e-05 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.868 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.630136945, 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039421115999999999, 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:51.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141379+0000", 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142797+0000", 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001418702 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142816+0000", 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8907999999999999e-05 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142825+0000", 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1020000000000003e-06 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142872+0000", 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6168999999999998e-05 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142891+0000", 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9561000000000001e-05 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180704+0000", 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037812643 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180801+0000", 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7361999999999994e-05 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2280 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'105, mlcod=73'100)", 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.609455667999999, 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029657706999999998, 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162060+0000", 2024-04-20T02:51:51.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162455+0000", 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039569799999999999 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162472+0000", 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6820000000000002e-05 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162485+0000", 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2388999999999999e-05 2024-04-20T02:51:51.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163758+0000", 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001273906 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163784+0000", 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5352000000000001e-05 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191663+0000", 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027879583999999999 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191719+0000", 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640000000000003e-05 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2559 3.0 e91/21 3:0870df8d:::348e4a2a55868b09db2e6d173951b4676cfef7376c5a81f7a82bb8f7ccc4ada5:head v 91'132, mlcod=91'130)", 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.302699907, 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032176705999999999, 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:51.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:51.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468815+0000", 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469831+0000", 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001015076 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469853+0000", 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2713000000000001e-05 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469865+0000", 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1904e-05 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469910+0000", 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5114999999999997e-05 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469934+0000", 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4133000000000001e-05 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.500951+0000", 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031016851000000002 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.500994+0000", 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2496000000000001e-05 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:51.881 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:51.882 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:51.883 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:51.884 INFO:tasks.rados.rados.0.smithi105.stdout:1272: expect (ObjNum 204 snap 19 seq_num 204) 2024-04-20T02:51:51.884 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:51.883+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:51.983 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:51.984+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:52.053 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:52.053 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:52.053 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:52.053 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:52.056 INFO:tasks.rados.rados.0.smithi105.stdout:1275: got (22) Invalid argument 2024-04-20T02:51:52.056 INFO:tasks.rados.rados.0.smithi105.stdout:1274: finishing write tid 1 to smithi10548823-21 2024-04-20T02:51:52.056 INFO:tasks.rados.rados.0.smithi105.stdout:1274: finishing write tid 2 to smithi10548823-21 2024-04-20T02:51:52.056 INFO:tasks.rados.rados.0.smithi105.stdout:1274: finishing write tid 3 to smithi10548823-21 2024-04-20T02:51:52.056 INFO:tasks.rados.rados.0.smithi105.stdout:1274: finishing write tid 4 to smithi10548823-21 2024-04-20T02:51:52.056 INFO:tasks.rados.rados.0.smithi105.stdout:1271: done (6 left) 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1272: done (5 left) 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1273: done (4 left) 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1275: done (3 left) 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1278: tier_promote oid 41 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1274: finishing write tid 5 to smithi10548823-21 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1274: oid 21 updating version 0 to 97 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1274: oid 21 version 97 is already newer than 95 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1274: oid 21 version 97 is already newer than 96 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1274: oid 21 version 97 is already newer than 94 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 97 (ObjNum 227 snap 22 seq_num 227) dirty exists 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1274: left oid 21 (ObjNum 227 snap 22 seq_num 227) 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1277: got (22) Invalid argument 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1276: finishing set_chunk to oid 39 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1274: done (3 left) 2024-04-20T02:51:52.057 INFO:tasks.rados.rados.0.smithi105.stdout:1276: done (2 left) 2024-04-20T02:51:52.058 INFO:tasks.rados.rados.0.smithi105.stdout:1277: done (1 left) 2024-04-20T02:51:52.058 INFO:tasks.rados.rados.0.smithi105.stdout:1279: read oid 8 snap -1 2024-04-20T02:51:52.058 INFO:tasks.rados.rados.0.smithi105.stdout:1279: expect (ObjNum 204 snap 19 seq_num 204) 2024-04-20T02:51:52.058 INFO:tasks.rados.rados.0.smithi105.stdout:1280: write (excl) oid 9 current snap is 22 2024-04-20T02:51:52.058 INFO:tasks.rados.rados.0.smithi105.stdout:1280: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-04-20T02:51:52.062 INFO:tasks.rados.rados.0.smithi105.stdout:1280: writing smithi10548823-9 from 631996 to 1163005 tid 1 2024-04-20T02:51:52.068 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:52.069 INFO:tasks.rados.rados.0.smithi105.stdout:1280: writing smithi10548823-9 from 1792365 to 2247101 tid 2 2024-04-20T02:51:52.075 INFO:tasks.rados.rados.0.smithi105.stdout:1280: writing smithi10548823-9 from 2761854 to 3433167 tid 3 2024-04-20T02:51:52.076 INFO:tasks.rados.rados.0.smithi105.stdout:1278: got (2) No such file or directory 2024-04-20T02:51:52.076 INFO:tasks.rados.rados.0.smithi105.stdout:1278: got expected ENOENT (src dne) 2024-04-20T02:51:52.076 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 206 (ObjNum 97 snap 0 seq_num 0) clean dne 2024-04-20T02:51:52.076 INFO:tasks.rados.rados.0.smithi105.stdout:1279: expect (ObjNum 204 snap 19 seq_num 204) 2024-04-20T02:51:52.087 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:52.087+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:52.140 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:51:52.140 INFO:teuthology.orchestra.run.smithi105.stderr:set pool 4 pg_num to 26 2024-04-20T02:51:52.188 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:52.189+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:52.204 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:52.205 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:52.205 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:52.205 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:52.205 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:52.216 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:52.280 INFO:tasks.rados.rados.0.smithi105.stdout:1280: finishing write tid 1 to smithi10548823-9 2024-04-20T02:51:52.280 INFO:tasks.rados.rados.0.smithi105.stdout:1280: finishing write tid 2 to smithi10548823-9 2024-04-20T02:51:52.280 INFO:tasks.rados.rados.0.smithi105.stdout:1281: read oid 1 snap -1 2024-04-20T02:51:52.280 INFO:tasks.rados.rados.0.smithi105.stdout:1281: expect deleted 2024-04-20T02:51:52.280 INFO:tasks.rados.rados.0.smithi105.stdout:1278: done (3 left) 2024-04-20T02:51:52.280 INFO:tasks.rados.rados.0.smithi105.stdout:1279: done (2 left) 2024-04-20T02:51:52.280 INFO:tasks.rados.rados.0.smithi105.stdout:1282: set_chunk oid 13 target oid 2024-04-20T02:51:52.288 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:52.289+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:52.309 INFO:tasks.rados.rados.0.smithi105.stdout:1282: set_chunk oid 13 offset: 229376 length: 1200640 target oid bf60052015bd9ca00a4d61cd4ab2824dc07855abe90adc4f6dbdb76578068933 offset: 0 snap -1 2024-04-20T02:51:52.309 INFO:tasks.rados.rados.0.smithi105.stdout:1283: rollback oid 5 current snap is 22 2024-04-20T02:51:52.309 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 5 to 21 2024-04-20T02:51:52.309 INFO:tasks.rados.rados.0.smithi105.stdout:1280: finishing write tid 3 to smithi10548823-9 2024-04-20T02:51:52.309 INFO:tasks.rados.rados.0.smithi105.stdout:1280: finishing write tid 4 to smithi10548823-9 2024-04-20T02:51:52.309 INFO:tasks.rados.rados.0.smithi105.stdout:1280: finishing write tid 5 to smithi10548823-9 2024-04-20T02:51:52.309 INFO:tasks.rados.rados.0.smithi105.stdout:1280: oid 9 updating version 0 to 230 2024-04-20T02:51:52.309 INFO:tasks.rados.rados.0.smithi105.stdout:1280: oid 9 version 230 is already newer than 229 2024-04-20T02:51:52.310 INFO:tasks.rados.rados.0.smithi105.stdout:1280: oid 9 updating version 230 to 232 2024-04-20T02:51:52.310 INFO:tasks.rados.rados.0.smithi105.stdout:1280: oid 9 version 232 is already newer than 231 2024-04-20T02:51:52.310 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 232 (ObjNum 228 snap 22 seq_num 228) dirty exists 2024-04-20T02:51:52.310 INFO:tasks.rados.rados.0.smithi105.stdout:1280: left oid 9 (ObjNum 228 snap 22 seq_num 228) 2024-04-20T02:51:52.310 INFO:tasks.rados.rados.0.smithi105.stdout:1280: done (3 left) 2024-04-20T02:51:52.310 INFO:tasks.rados.rados.0.smithi105.stdout:1281: done (2 left) 2024-04-20T02:51:52.310 INFO:tasks.rados.rados.0.smithi105.stdout:1284: set_chunk oid 35 target oid 2024-04-20T02:51:52.339 INFO:tasks.rados.rados.0.smithi105.stdout:1284: set_chunk oid 35 offset: 163328 length: 1788928 target oid dfb4328d4bfe97da25177f5a4bc81e390ac42ee49ff52a9e0347a6a132c60b44 offset: 0 snap -1 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:1285: tier_flush oid 16 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:1285: tier_flush oid 16 snap -1 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:1282: finishing set_chunk to oid 13 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 142 (ObjNum 151 snap 10 seq_num 151) dirty exists 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:1282: done (3 left) 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:1286: write oid 39 current snap is 22 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:1283: finishing rollback tid 0 to smithi10548823-5 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:1283: finishing rollback tid 1 to smithi10548823-5 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:1283: finishing rollback tid 2 to smithi10548823-5 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 155 (ObjNum 186 snap 18 seq_num 186) dirty exists 2024-04-20T02:51:52.340 INFO:tasks.rados.rados.0.smithi105.stdout:1286: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-04-20T02:51:52.346 INFO:tasks.rados.rados.0.smithi105.stdout:1286: writing smithi10548823-39 from 648803 to 1371414 tid 1 2024-04-20T02:51:52.355 INFO:tasks.rados.rados.0.smithi105.stdout:1286: writing smithi10548823-39 from 1967200 to 2681947 tid 2 2024-04-20T02:51:52.362 INFO:tasks.rados.rados.0.smithi105.stdout:1286: writing smithi10548823-39 from 3421983 to 3848803 tid 3 2024-04-20T02:51:52.362 INFO:tasks.rados.rados.0.smithi105.stdout:1283: done (3 left) 2024-04-20T02:51:52.362 INFO:tasks.rados.rados.0.smithi105.stdout:1287: delete oid 36 current snap is 22 2024-04-20T02:51:52.363 INFO:tasks.rados.rados.0.smithi105.stdout:1285: got (2) No such file or directory 2024-04-20T02:51:52.363 INFO:tasks.rados.rados.0.smithi105.stdout:1285: got expected ENOENT (src dne) 2024-04-20T02:51:52.363 INFO:tasks.rados.rados.0.smithi105.stdout:1284: finishing set_chunk to oid 35 2024-04-20T02:51:52.363 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 163328 length: 1788928 2024-04-20T02:51:52.363 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 124 snap 9 seq_num 124) dirty exists 2024-04-20T02:51:52.367 INFO:tasks.rados.rados.0.smithi105.stdout:1284: done (3 left) 2024-04-20T02:51:52.367 INFO:tasks.rados.rados.0.smithi105.stdout:1285: done (2 left) 2024-04-20T02:51:52.367 INFO:tasks.rados.rados.0.smithi105.stdout:1287: done (1 left) 2024-04-20T02:51:52.367 INFO:tasks.rados.rados.0.smithi105.stdout:1288: write oid 15 current snap is 22 2024-04-20T02:51:52.367 INFO:tasks.rados.rados.0.smithi105.stdout:1288: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-04-20T02:51:52.373 INFO:tasks.rados.rados.0.smithi105.stdout:1288: writing smithi10548823-15 from 682417 to 1155526 tid 1 2024-04-20T02:51:52.379 INFO:tasks.rados.rados.0.smithi105.stdout:1288: writing smithi10548823-15 from 1567811 to 2169874 tid 2 2024-04-20T02:51:52.388 INFO:tasks.rados.rados.0.smithi105.stdout:1288: writing smithi10548823-15 from 2844123 to 3465605 tid 3 2024-04-20T02:51:52.388 INFO:tasks.rados.rados.0.smithi105.stdout:1289: read oid 25 snap 21 2024-04-20T02:51:52.388 INFO:tasks.rados.rados.0.smithi105.stdout:1289: expect (ObjNum 169 snap 14 seq_num 169) 2024-04-20T02:51:52.388 INFO:tasks.rados.rados.0.smithi105.stdout:1290: read oid 21 snap -1 2024-04-20T02:51:52.388 INFO:tasks.rados.rados.0.smithi105.stdout:1290: expect (ObjNum 227 snap 22 seq_num 227) 2024-04-20T02:51:52.388 INFO:tasks.rados.rados.0.smithi105.stdout:1291: set_chunk oid 17 target oid 2024-04-20T02:51:52.391 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:52.389+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:52.398 INFO:tasks.rados.rados.0.smithi105.stdout:1291: set_chunk oid 17 offset: 2379776 length: 176128 target oid 40aca62fbe765bd4ae0726f6f469fa6cc5f8efecf09db71cc1eb791aaf0a0191 offset: 0 snap -1 2024-04-20T02:51:52.398 INFO:tasks.rados.rados.0.smithi105.stdout:1292: write (excl) oid 46 current snap is 22 2024-04-20T02:51:52.398 INFO:tasks.rados.rados.0.smithi105.stdout:1292: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-04-20T02:51:52.401 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:52.401 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:52.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:52.401 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:52.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.401 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.855108197999996, 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:52.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:52.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:51:52.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:51:52.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:51:52.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.848887124000001, 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.409 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:51:52.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:51:52.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:52.415 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.840486920000004, 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:51:52.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:52.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:51:52.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:971 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (56) in=7b,truncate 3277332] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.840168851000001, 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051333776999999997, 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.422 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 971 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558679+0000", 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558678+0000", 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558686+0000", 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7330000000000003e-06 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558693+0000", 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.866e-06 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558698+0000", 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.49e-06 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594685+0000", 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035986753000000003 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594792+0000", 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000107129 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594958+0000", 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000165429 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595721+0000", 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00076360100000000004 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.596242+0000", 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052083800000000003 2024-04-20T02:51:52.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609915+0000", 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013673431 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609958+0000", 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2759000000000002e-05 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610013+0000", 2024-04-20T02:51:52.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4421e-05 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.840144275, 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.428 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:51:52.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:51:52.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.033822272000002, 2024-04-20T02:51:52.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:52.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:51:52.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.026559310000003, 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:51:52.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:51:52.445 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.020616781000001, 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.446 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:51:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:51:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:51:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:51:52.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.020413789000003, 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:51:52.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:52.460 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.020390861999999, 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 58.010307128000001, 2024-04-20T02:51:52.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:51:52.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:51:52.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.469 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.993421720000001, 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:52.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.977145278999998, 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.474 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:51:52.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:51:52.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.952589807000003, 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:51:52.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:51:52.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:51:52.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.638274031000002, 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:51:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:51:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2046 4.11 4:8b927dd6:::smithi10548823-28:head [write 2025476~459397] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.767100+0000", 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.631747997000002, 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050129193000000002, 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2046 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767100+0000", 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767100+0000", 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767098+0000", 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:52.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767266+0000", 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000167901 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767281+0000", 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5478999999999999e-05 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.767294+0000", 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3259e-05 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.786324+0000", 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019029404 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.786421+0000", 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6897000000000004e-05 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.786579+0000", 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015888299999999999 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804648+0000", 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018068208999999998 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804685+0000", 2024-04-20T02:51:52.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7832e-05 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817139+0000", 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012453862 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817184+0000", 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4437000000000002e-05 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817229+0000", 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5136000000000003e-05 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2048 4.11 4:8b927dd6:::smithi10548823-28:head [setxattr _header (57) in=7b,truncate 3042067] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.770221+0000", 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.628627042000002, 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049037753000000003, 2024-04-20T02:51:52.497 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2048 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770221+0000", 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770221+0000", 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770220+0000", 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770223+0000", 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.32e-06 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770230+0000", 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2759999999999997e-06 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770238+0000", 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2539999999999992e-06 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816299+0000", 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046060940000000002 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816426+0000", 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012759199999999999 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816612+0000", 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000185885 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818165+0000", 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001552138 2024-04-20T02:51:52.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.818192+0000", 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7271e-05 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819143+0000", 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00095093499999999997 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819199+0000", 2024-04-20T02:51:52.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6334e-05 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819258+0000", 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9373000000000002e-05 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2049 4.11 4:8b927dd6:::smithi10548823-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.628605256, 2024-04-20T02:51:52.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049230470999999998, 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2049 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:52.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770242+0000", 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770245+0000", 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4430000000000002e-06 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770251+0000", 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2929999999999997e-06 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.770260+0000", 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2930000000000006e-06 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.816910+0000", 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046650129999999998 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817004+0000", 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3665999999999994e-05 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819289+0000", 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002284847 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819379+0000", 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0274999999999999e-05 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.819473+0000", 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3882999999999995e-05 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.508 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2050 4.0 4:007b2a87:::smithi10548823-39:head [write 658874~771437,stat] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.778080+0000", 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.620768182999999, 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051190877000000003, 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2050 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778080+0000", 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778080+0000", 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778077+0000", 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778477+0000", 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039974900000000002 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778496+0000", 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9335e-05 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778510+0000", 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3377000000000001e-05 2024-04-20T02:51:52.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778572+0000", 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1898999999999998e-05 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778685+0000", 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011372800000000001 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.778917+0000", 2024-04-20T02:51:52.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195799999999999 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817646+0000", 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038729087000000002 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817828+0000", 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181986 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.829157+0000", 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011328379 2024-04-20T02:51:52.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.829216+0000", 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8962000000000002e-05 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.829270+0000", 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4727e-05 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2545 4.9 e91/81 4:95e19a88:::smithi10548823-40:head v 91'253, mlcod=91'251)", 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.983636916, 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054757404000000003, 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415210+0000", 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416004+0000", 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079381199999999997 2024-04-20T02:51:52.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416022+0000", 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7969e-05 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416031+0000", 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9279999999999999e-06 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416100+0000", 2024-04-20T02:51:52.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8891000000000002e-05 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416128+0000", 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8203000000000001e-05 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469912+0000", 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053783794000000003 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:52.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:52.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469968+0000", 2024-04-20T02:51:52.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6474000000000002e-05 2024-04-20T02:51:52.519 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.519 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.519 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.519 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:52.519 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:52.519 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:52.521 INFO:tasks.rados.rados.0.smithi105.stdout:1292: writing smithi10548823-46 from 699224 to 1247582 tid 1 2024-04-20T02:51:52.521 INFO:tasks.rados.rados.0.smithi105.stdout:1292: writing smithi10548823-46 from 2026293 to 2488367 tid 2 2024-04-20T02:51:52.522 INFO:tasks.rados.rados.0.smithi105.stdout:1292: writing smithi10548823-46 from 3271546 to 3704300 tid 3 2024-04-20T02:51:52.522 INFO:tasks.rados.rados.0.smithi105.stdout:1293: read oid 18 snap -1 2024-04-20T02:51:52.522 INFO:tasks.rados.rados.0.smithi105.stdout:1293: expect deleted 2024-04-20T02:51:52.522 INFO:tasks.rados.rados.0.smithi105.stdout:1294: read oid 30 snap 21 2024-04-20T02:51:52.522 INFO:tasks.rados.rados.0.smithi105.stdout:1294: expect (ObjNum 206 snap 19 seq_num 206) 2024-04-20T02:51:52.522 INFO:tasks.rados.rados.0.smithi105.stdout:1295: read oid 8 snap -1 2024-04-20T02:51:52.522 INFO:tasks.rados.rados.0.smithi105.stdout:1295: expect (ObjNum 204 snap 19 seq_num 204) 2024-04-20T02:51:52.522 INFO:tasks.rados.rados.0.smithi105.stdout:1296: write (excl) oid 19 current snap is 22 2024-04-20T02:51:52.522 INFO:tasks.rados.rados.0.smithi105.stdout:1296: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-04-20T02:51:52.523 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:52.524 INFO:tasks.rados.rados.0.smithi105.stdout:1296: writing smithi10548823-19 from 716031 to 1339638 tid 1 2024-04-20T02:51:52.524 INFO:tasks.rados.rados.0.smithi105.stdout:1296: writing smithi10548823-19 from 2084775 to 2690507 tid 2 2024-04-20T02:51:52.524 INFO:tasks.rados.rados.0.smithi105.stdout:1296: writing smithi10548823-19 from 3298969 to 3916031 tid 3 2024-04-20T02:51:52.524 INFO:tasks.rados.rados.0.smithi105.stdout:1297: tier_evict oid 11 2024-04-20T02:51:52.524 INFO:tasks.rados.rados.0.smithi105.stdout:1297: tier_evict oid 11 snap -1 2024-04-20T02:51:52.524 INFO:tasks.rados.rados.0.smithi105.stdout:1289: expect (ObjNum 169 snap 14 seq_num 169) 2024-04-20T02:51:52.525 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:52.521+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:52.593 INFO:tasks.rados.rados.0.smithi105.stdout:1291: finishing set_chunk to oid 17 2024-04-20T02:51:52.594 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 71 (ObjNum 216 snap 21 seq_num 216) dirty exists 2024-04-20T02:51:52.594 INFO:tasks.rados.rados.0.smithi105.stdout:1286: finishing write tid 1 to smithi10548823-39 2024-04-20T02:51:52.594 INFO:tasks.rados.rados.0.smithi105.stdout:1286: finishing write tid 2 to smithi10548823-39 2024-04-20T02:51:52.594 INFO:tasks.rados.rados.0.smithi105.stdout:1289: done (10 left) 2024-04-20T02:51:52.594 INFO:tasks.rados.rados.0.smithi105.stdout:1291: done (9 left) 2024-04-20T02:51:52.594 INFO:tasks.rados.rados.0.smithi105.stdout:1298: set_chunk oid 47 target oid 2024-04-20T02:51:52.603 INFO:tasks.rados.rados.0.smithi105.stdout:1298: set_chunk oid 47 offset: 885760 length: 293376 target oid ba66664d949b4034e80eb872f82b65af5f9054a1379f16e1690baf0bf20768b4 offset: 0 snap 21 2024-04-20T02:51:52.603 INFO:tasks.rados.rados.0.smithi105.stdout:1299: write oid 24 current snap is 22 2024-04-20T02:51:52.603 INFO:tasks.rados.rados.0.smithi105.stdout:1286: finishing write tid 3 to smithi10548823-39 2024-04-20T02:51:52.603 INFO:tasks.rados.rados.0.smithi105.stdout:1290: expect (ObjNum 227 snap 22 seq_num 227) 2024-04-20T02:51:52.621 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:52.621+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:52.654 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:52.654 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:52.654 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:52.654 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:52.664 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:52.723 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:52.723+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:52.781 INFO:tasks.rados.rados.0.smithi105.stdout:1288: finishing write tid 1 to smithi10548823-15 2024-04-20T02:51:52.781 INFO:tasks.rados.rados.0.smithi105.stdout:1286: finishing write tid 4 to smithi10548823-39 2024-04-20T02:51:52.781 INFO:tasks.rados.rados.0.smithi105.stdout:1286: finishing write tid 5 to smithi10548823-39 2024-04-20T02:51:52.781 INFO:tasks.rados.rados.0.smithi105.stdout:1286: oid 39 updating version 0 to 103 2024-04-20T02:51:52.781 INFO:tasks.rados.rados.0.smithi105.stdout:1286: oid 39 version 103 is already newer than 102 2024-04-20T02:51:52.781 INFO:tasks.rados.rados.0.smithi105.stdout:1286: oid 39 version 103 is already newer than 100 2024-04-20T02:51:52.782 INFO:tasks.rados.rados.0.smithi105.stdout:1286: oid 39 version 103 is already newer than 101 2024-04-20T02:51:52.782 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 103 (ObjNum 229 snap 22 seq_num 229) dirty exists 2024-04-20T02:51:52.782 INFO:tasks.rados.rados.0.smithi105.stdout:1286: left oid 39 (ObjNum 229 snap 22 seq_num 229) 2024-04-20T02:51:52.782 INFO:tasks.rados.rados.0.smithi105.stdout:1288: finishing write tid 2 to smithi10548823-15 2024-04-20T02:51:52.782 INFO:tasks.rados.rados.0.smithi105.stdout:1299: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-04-20T02:51:52.788 INFO:tasks.rados.rados.0.smithi105.stdout:1299: writing smithi10548823-24 from 732838 to 1431694 tid 1 2024-04-20T02:51:52.794 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:52.794 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:52.794 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:52.795 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:52.795 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:52.796 INFO:tasks.rados.rados.0.smithi105.stdout:1299: writing smithi10548823-24 from 2026904 to 2492647 tid 2 2024-04-20T02:51:52.804 INFO:tasks.rados.rados.0.smithi105.stdout:1299: writing smithi10548823-24 from 3210039 to 3781690 tid 3 2024-04-20T02:51:52.804 INFO:tasks.rados.rados.0.smithi105.stdout:1286: done (10 left) 2024-04-20T02:51:52.804 INFO:tasks.rados.rados.0.smithi105.stdout:1290: done (9 left) 2024-04-20T02:51:52.804 INFO:tasks.rados.rados.0.smithi105.stdout:1293: done (8 left) 2024-04-20T02:51:52.804 INFO:tasks.rados.rados.0.smithi105.stdout:1300: read oid 48 snap 21 2024-04-20T02:51:52.804 INFO:tasks.rados.rados.0.smithi105.stdout:1300: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:51:52.804 INFO:tasks.rados.rados.0.smithi105.stdout:1301: read oid 29 snap -1 2024-04-20T02:51:52.804 INFO:tasks.rados.rados.0.smithi105.stdout:1301: expect (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:51:52.805 INFO:tasks.rados.rados.0.smithi105.stdout:1295: expect (ObjNum 204 snap 19 seq_num 204) 2024-04-20T02:51:52.806 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:52.849 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:52.850+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:52.924 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:52.924+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:52.937 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:52.937 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:52.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:52.937 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:52.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.937 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:52.937 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.993118561000003, 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:52.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:52.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:51:52.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:51:52.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.985640242999999, 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:51:52.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:51:52.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:52.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1146 4.d 4:b376c20e:::smithi10548823-49:head [write 3437750~125301] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:52.951 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.981072382000001, 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070707487999999999, 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1146 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955023+0000", 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955022+0000", 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:52.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955179+0000", 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015657 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955188+0000", 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.645e-06 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955195+0000", 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2910000000000004e-06 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978329+0000", 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02313401 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978450+0000", 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121438 2024-04-20T02:51:52.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.978652+0000", 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020163700000000001 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984380+0000", 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057282510000000002 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991325+0000", 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069446830000000001 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025680+0000", 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034355284 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025709+0000", 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8781000000000001e-05 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025731+0000", 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1963e-05 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1147 4.d 4:b376c20e:::smithi10548823-49:head [setxattr _header (56) in=7b,truncate 3563051] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.980894044999999, 2024-04-20T02:51:52.958 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070713812000000001, 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1147 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955202+0000", 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955201+0000", 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:52.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955204+0000", 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.05e-06 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955207+0000", 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1949999999999999e-06 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955210+0000", 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983351+0000", 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028140978000000001 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983488+0000", 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013647 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983668+0000", 2024-04-20T02:51:52.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000180335 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984487+0000", 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081847099999999998 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.991393+0000", 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069068569999999998 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025852+0000", 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034458946999999997 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025883+0000", 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0261e-05 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025915+0000", 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2654999999999998e-05 2024-04-20T02:51:52.964 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.980882344000001, 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:52.965 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:51:52.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:51:52.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.970 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.557453410000001, 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.971 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:52.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:52.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:51:52.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:52.978 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.557416527000001, 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:52.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:51:52.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:51:52.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:52.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:51:52.985 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.557391893999998, 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:52.986 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:52.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:51:52.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.550898482999997, 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:52.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:51:52.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:51:52.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:51:52.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:51:52.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.544834461000001, 2024-04-20T02:51:52.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:53.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:51:53.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:51:53.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.534772695999997, 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:51:53.006 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:51:53.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:51:53.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.534080091999996, 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.013 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:53.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:51:53.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.534068437999998, 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.020 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:51:53.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:51:53.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.025 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.828003881999997, 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.026 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:51:53.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1426 4.1 4:8b927dd6:::smithi10548823-28:head [write 1331027~582033] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.816449720000001, 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068185170000000003, 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1426 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.031 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119646+0000", 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.119645+0000", 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120235+0000", 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059073600000000002 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120249+0000", 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4168000000000001e-05 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120258+0000", 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.2719999999999997e-06 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124047+0000", 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003788839 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124154+0000", 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000107747 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.124307+0000", 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000152998 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148497+0000", 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024189789999999999 2024-04-20T02:51:53.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168007+0000", 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019509463000000001 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187765+0000", 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019758738000000001 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:53.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187802+0000", 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7153999999999997e-05 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187831+0000", 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8592999999999999e-05 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1428 4.1 4:8b927dd6:::smithi10548823-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:53.037 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.815796890999998, 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067836666000000004, 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1428 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120299+0000", 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120298+0000", 2024-04-20T02:51:53.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120301+0000", 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.853e-06 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120308+0000", 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0559999999999999e-06 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.120313+0000", 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1680000000000003e-06 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148583+0000", 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028269882 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.148673+0000", 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9922999999999999e-05 2024-04-20T02:51:53.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.187934+0000", 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039260647000000003 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188031+0000", 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7468999999999998e-05 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188135+0000", 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104126 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.808943165999999, 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:51:53.043 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:51:53.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:51:53.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.802996852, 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:51:53.050 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:53.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:51:53.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.802431411000001, 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.057 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:51:53.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:51:53.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:51:53.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:51:53.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:53.067 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.802405380000003, 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:53.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:51:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:51:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:51:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:53.073 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:53.074 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:53.076 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:53.076 INFO:tasks.rados.rados.0.smithi105.stdout:1288: finishing write tid 3 to smithi10548823-15 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:1288: finishing write tid 4 to smithi10548823-15 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:1288: finishing write tid 5 to smithi10548823-15 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:1288: oid 15 updating version 0 to 266 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:1288: oid 15 updating version 266 to 268 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:1288: oid 15 version 268 is already newer than 265 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:1288: oid 15 version 268 is already newer than 267 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 268 (ObjNum 231 snap 22 seq_num 231) dirty exists 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:1288: left oid 15 (ObjNum 231 snap 22 seq_num 231) 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:1297: got (22) Invalid argument 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:1292: finishing write tid 1 to smithi10548823-46 2024-04-20T02:51:53.077 INFO:tasks.rados.rados.0.smithi105.stdout:1302: read oid 22 snap -1 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1302: expect (ObjNum 225 snap 22 seq_num 225) 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1288: done (10 left) 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1295: done (9 left) 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1297: done (8 left) 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1303: write oid 39 current snap is 22 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1292: finishing write tid 2 to smithi10548823-46 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1292: finishing write tid 3 to smithi10548823-46 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1292: finishing write tid 4 to smithi10548823-46 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1292: finishing write tid 5 to smithi10548823-46 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1292: oid 46 updating version 0 to 81 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1292: oid 46 updating version 81 to 82 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1292: oid 46 updating version 82 to 83 2024-04-20T02:51:53.078 INFO:tasks.rados.rados.0.smithi105.stdout:1292: oid 46 version 83 is already newer than 80 2024-04-20T02:51:53.079 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 83 (ObjNum 232 snap 22 seq_num 232) dirty exists 2024-04-20T02:51:53.079 INFO:tasks.rados.rados.0.smithi105.stdout:1292: left oid 46 (ObjNum 232 snap 22 seq_num 232) 2024-04-20T02:51:53.079 INFO:tasks.rados.rados.0.smithi105.stdout:1294: expect (ObjNum 206 snap 19 seq_num 206) 2024-04-20T02:51:53.079 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:53.074+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:53.175 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:53.175+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:53.223 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:53.241 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:53.245 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:53.246 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:53.246 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:53.246 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:53.256 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:53.261 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:53.276 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:53.276+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:53.281 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1296: finishing write tid 1 to smithi10548823-19 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1296: finishing write tid 2 to smithi10548823-19 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1296: finishing write tid 3 to smithi10548823-19 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1296: finishing write tid 4 to smithi10548823-19 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1296: finishing write tid 5 to smithi10548823-19 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1296: oid 19 updating version 0 to 105 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1296: oid 19 version 105 is already newer than 104 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1296: oid 19 updating version 105 to 106 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1296: oid 19 updating version 106 to 107 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 107 (ObjNum 233 snap 22 seq_num 233) dirty exists 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1296: left oid 19 (ObjNum 233 snap 22 seq_num 233) 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1298: finishing set_chunk to oid 47 2024-04-20T02:51:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 885760 length: 293376 2024-04-20T02:51:53.288 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:51:53.288 INFO:tasks.rados.rados.0.smithi105.stdout:1301: expect (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:51:53.300 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:53.320 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:53.339 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:53.358 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:53.359 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:53.360 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:53.360 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:53.360 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:53.360 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:53.360 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:53.360 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:53.360 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:53.376 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:53.377+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:53.427 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:53.427 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:53.437 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:53.478 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:53.478+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:53.522 INFO:tasks.rados.rados.0.smithi105.stdout:1300: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:51:53.579 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:53.580+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:53.606 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:53.606 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:53.616 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:53.680 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:53.680+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:53.729 INFO:tasks.rados.rados.0.smithi105.stdout:1299: finishing write tid 1 to smithi10548823-24 2024-04-20T02:51:53.729 INFO:tasks.rados.rados.0.smithi105.stdout:1299: finishing write tid 2 to smithi10548823-24 2024-04-20T02:51:53.729 INFO:tasks.rados.rados.0.smithi105.stdout:1299: finishing write tid 3 to smithi10548823-24 2024-04-20T02:51:53.730 INFO:tasks.rados.rados.0.smithi105.stdout:1299: finishing write tid 4 to smithi10548823-24 2024-04-20T02:51:53.730 INFO:tasks.rados.rados.0.smithi105.stdout:1299: finishing write tid 5 to smithi10548823-24 2024-04-20T02:51:53.730 INFO:tasks.rados.rados.0.smithi105.stdout:1299: oid 24 updating version 0 to 271 2024-04-20T02:51:53.730 INFO:tasks.rados.rados.0.smithi105.stdout:1299: oid 24 updating version 271 to 273 2024-04-20T02:51:53.730 INFO:tasks.rados.rados.0.smithi105.stdout:1299: oid 24 version 273 is already newer than 270 2024-04-20T02:51:53.730 INFO:tasks.rados.rados.0.smithi105.stdout:1299: oid 24 version 273 is already newer than 272 2024-04-20T02:51:53.730 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 273 (ObjNum 234 snap 22 seq_num 234) dirty exists 2024-04-20T02:51:53.730 INFO:tasks.rados.rados.0.smithi105.stdout:1299: left oid 24 (ObjNum 234 snap 22 seq_num 234) 2024-04-20T02:51:53.730 INFO:tasks.rados.rados.0.smithi105.stdout:1303: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-04-20T02:51:53.737 INFO:tasks.rados.rados.0.smithi105.stdout:1303: writing smithi10548823-39 from 749645 to 1523750 tid 1 2024-04-20T02:51:53.743 INFO:tasks.rados.rados.0.smithi105.stdout:1303: writing smithi10548823-39 from 2085386 to 2694787 tid 2 2024-04-20T02:51:53.751 INFO:tasks.rados.rados.0.smithi105.stdout:1303: writing smithi10548823-39 from 3237462 to 3949645 tid 3 2024-04-20T02:51:53.751 INFO:tasks.rados.rados.0.smithi105.stdout:1292: done (8 left) 2024-04-20T02:51:53.751 INFO:tasks.rados.rados.0.smithi105.stdout:1294: done (7 left) 2024-04-20T02:51:53.751 INFO:tasks.rados.rados.0.smithi105.stdout:1296: done (6 left) 2024-04-20T02:51:53.751 INFO:tasks.rados.rados.0.smithi105.stdout:1298: done (5 left) 2024-04-20T02:51:53.751 INFO:tasks.rados.rados.0.smithi105.stdout:1299: done (4 left) 2024-04-20T02:51:53.752 INFO:tasks.rados.rados.0.smithi105.stdout:1300: done (3 left) 2024-04-20T02:51:53.752 INFO:tasks.rados.rados.0.smithi105.stdout:1301: done (2 left) 2024-04-20T02:51:53.752 INFO:tasks.rados.rados.0.smithi105.stdout:1304: read oid 28 snap -1 2024-04-20T02:51:53.752 INFO:tasks.rados.rados.0.smithi105.stdout:1304: expect (ObjNum 200 snap 19 seq_num 200) 2024-04-20T02:51:53.752 INFO:tasks.rados.rados.0.smithi105.stdout:1305: delete oid 17 current snap is 22 2024-04-20T02:51:53.752 INFO:tasks.rados.rados.0.smithi105.stdout:1302: expect (ObjNum 225 snap 22 seq_num 225) 2024-04-20T02:51:53.781 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:53.782+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:53.801 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:53.801 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:53.812 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:53.896 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:53.897+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:53.967 INFO:tasks.rados.rados.0.smithi105.stdout:1304: expect (ObjNum 200 snap 19 seq_num 200) 2024-04-20T02:51:53.980 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:53.980 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:53.983 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:53.984+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:53.990 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:54.084 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:54.085+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:54.154 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:51:54.159 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:54.159 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:54.169 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:54.198 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:54.198+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:54.223 INFO:tasks.rados.rados.0.smithi105.stdout:1303: finishing write tid 1 to smithi10548823-39 2024-04-20T02:51:54.223 INFO:tasks.rados.rados.0.smithi105.stdout:1303: finishing write tid 2 to smithi10548823-39 2024-04-20T02:51:54.223 INFO:tasks.rados.rados.0.smithi105.stdout:1303: finishing write tid 3 to smithi10548823-39 2024-04-20T02:51:54.223 INFO:tasks.rados.rados.0.smithi105.stdout:1303: finishing write tid 4 to smithi10548823-39 2024-04-20T02:51:54.223 INFO:tasks.rados.rados.0.smithi105.stdout:1302: done (3 left) 2024-04-20T02:51:54.223 INFO:tasks.rados.rados.0.smithi105.stdout:1304: done (2 left) 2024-04-20T02:51:54.223 INFO:tasks.rados.rados.0.smithi105.stdout:1305: done (1 left) 2024-04-20T02:51:54.224 INFO:tasks.rados.rados.0.smithi105.stdout:1306: write (excl) oid 18 current snap is 22 2024-04-20T02:51:54.224 INFO:tasks.rados.rados.0.smithi105.stdout:1303: finishing write tid 5 to smithi10548823-39 2024-04-20T02:51:54.224 INFO:tasks.rados.rados.0.smithi105.stdout:1303: oid 39 updating version 0 to 105 2024-04-20T02:51:54.224 INFO:tasks.rados.rados.0.smithi105.stdout:1303: oid 39 version 105 is already newer than 104 2024-04-20T02:51:54.224 INFO:tasks.rados.rados.0.smithi105.stdout:1303: oid 39 updating version 105 to 107 2024-04-20T02:51:54.224 INFO:tasks.rados.rados.0.smithi105.stdout:1303: oid 39 version 107 is already newer than 106 2024-04-20T02:51:54.224 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 107 (ObjNum 235 snap 22 seq_num 235) dirty exists 2024-04-20T02:51:54.224 INFO:tasks.rados.rados.0.smithi105.stdout:1303: left oid 39 (ObjNum 235 snap 22 seq_num 235) 2024-04-20T02:51:54.224 INFO:tasks.rados.rados.0.smithi105.stdout:1306: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-04-20T02:51:54.231 INFO:tasks.rados.rados.0.smithi105.stdout:1306: writing smithi10548823-18 from 783259 to 1424215 tid 1 2024-04-20T02:51:54.238 INFO:tasks.rados.rados.0.smithi105.stdout:1306: writing smithi10548823-18 from 1918703 to 2531773 tid 2 2024-04-20T02:51:54.244 INFO:tasks.rados.rados.0.smithi105.stdout:1306: writing smithi10548823-18 from 3008661 to 3414128 tid 3 2024-04-20T02:51:54.247 INFO:tasks.rados.rados.0.smithi105.stdout:1306: writing smithi10548823-18 from 3821939 to 3983259 tid 4 2024-04-20T02:51:54.248 INFO:tasks.rados.rados.0.smithi105.stdout:1303: done (1 left) 2024-04-20T02:51:54.248 INFO:tasks.rados.rados.0.smithi105.stdout:1307: write oid 31 current snap is 22 2024-04-20T02:51:54.248 INFO:tasks.rados.rados.0.smithi105.stdout:1307: seq_num 238 ranges {400066=716205,1860832=139234} 2024-04-20T02:51:54.253 INFO:tasks.rados.rados.0.smithi105.stdout:1307: writing smithi10548823-31 from 400066 to 1116271 tid 1 2024-04-20T02:51:54.259 INFO:tasks.rados.rados.0.smithi105.stdout:1307: writing smithi10548823-31 from 1860832 to 2000066 tid 2 2024-04-20T02:51:54.259 INFO:tasks.rados.rados.0.smithi105.stdout:1308: rollback oid 28 current snap is 22 2024-04-20T02:51:54.259 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 28 to 21 2024-04-20T02:51:54.259 INFO:tasks.rados.rados.0.smithi105.stdout:1309: read oid 42 snap -1 2024-04-20T02:51:54.259 INFO:tasks.rados.rados.0.smithi105.stdout:1309: expect (ObjNum 222 snap 21 seq_num 222) 2024-04-20T02:51:54.259 INFO:tasks.rados.rados.0.smithi105.stdout:1310: read oid 27 snap -1 2024-04-20T02:51:54.259 INFO:tasks.rados.rados.0.smithi105.stdout:1310: expect (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:51:54.260 INFO:tasks.rados.rados.0.smithi105.stdout:1311: delete oid 10 current snap is 22 2024-04-20T02:51:54.265 INFO:tasks.rados.rados.0.smithi105.stdout:1310: expect (ObjNum 103 snap 5 seq_num 103) 2024-04-20T02:51:54.292 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:54.290+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:54.361 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:54.361 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:54.371 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:54.415 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:54.416+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:54.483 INFO:tasks.rados.rados.0.smithi105.stdout:1308: finishing rollback tid 0 to smithi10548823-28 2024-04-20T02:51:54.483 INFO:tasks.rados.rados.0.smithi105.stdout:1306: finishing write tid 1 to smithi10548823-18 2024-04-20T02:51:54.483 INFO:tasks.rados.rados.0.smithi105.stdout:1308: finishing rollback tid 1 to smithi10548823-28 2024-04-20T02:51:54.483 INFO:tasks.rados.rados.0.smithi105.stdout:1308: finishing rollback tid 2 to smithi10548823-28 2024-04-20T02:51:54.483 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 159 (ObjNum 200 snap 19 seq_num 200) dirty exists 2024-04-20T02:51:54.483 INFO:tasks.rados.rados.0.smithi105.stdout:1307: finishing write tid 1 to smithi10548823-31 2024-04-20T02:51:54.483 INFO:tasks.rados.rados.0.smithi105.stdout:1308: done (5 left) 2024-04-20T02:51:54.483 INFO:tasks.rados.rados.0.smithi105.stdout:1310: done (4 left) 2024-04-20T02:51:54.484 INFO:tasks.rados.rados.0.smithi105.stdout:1311: done (3 left) 2024-04-20T02:51:54.484 INFO:tasks.rados.rados.0.smithi105.stdout:1312: read oid 19 snap 21 2024-04-20T02:51:54.484 INFO:tasks.rados.rados.0.smithi105.stdout:1312: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:54.484 INFO:tasks.rados.rados.0.smithi105.stdout:1313: write oid 12 current snap is 22 2024-04-20T02:51:54.484 INFO:tasks.rados.rados.0.smithi105.stdout:1307: finishing write tid 2 to smithi10548823-31 2024-04-20T02:51:54.484 INFO:tasks.rados.rados.0.smithi105.stdout:1307: finishing write tid 3 to smithi10548823-31 2024-04-20T02:51:54.484 INFO:tasks.rados.rados.0.smithi105.stdout:1307: finishing write tid 4 to smithi10548823-31 2024-04-20T02:51:54.484 INFO:tasks.rados.rados.0.smithi105.stdout:1307: oid 31 updating version 0 to 157 2024-04-20T02:51:54.484 INFO:tasks.rados.rados.0.smithi105.stdout:1307: oid 31 updating version 157 to 159 2024-04-20T02:51:54.484 INFO:tasks.rados.rados.0.smithi105.stdout:1307: oid 31 version 159 is already newer than 158 2024-04-20T02:51:54.485 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 159 (ObjNum 238 snap 22 seq_num 238) dirty exists 2024-04-20T02:51:54.485 INFO:tasks.rados.rados.0.smithi105.stdout:1307: left oid 31 (ObjNum 238 snap 22 seq_num 238) 2024-04-20T02:51:54.485 INFO:tasks.rados.rados.0.smithi105.stdout:1306: finishing write tid 2 to smithi10548823-18 2024-04-20T02:51:54.485 INFO:tasks.rados.rados.0.smithi105.stdout:1306: finishing write tid 3 to smithi10548823-18 2024-04-20T02:51:54.485 INFO:tasks.rados.rados.0.smithi105.stdout:1306: finishing write tid 4 to smithi10548823-18 2024-04-20T02:51:54.485 INFO:tasks.rados.rados.0.smithi105.stdout:1306: finishing write tid 5 to smithi10548823-18 2024-04-20T02:51:54.485 INFO:tasks.rados.rados.0.smithi105.stdout:1313: seq_num 240 ranges {433680=466703,1577796=455884} 2024-04-20T02:51:54.487 INFO:tasks.rados.rados.0.smithi105.stdout:1313: writing smithi10548823-12 from 433680 to 900383 tid 1 2024-04-20T02:51:54.487 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:54.487+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:54.493 INFO:tasks.rados.rados.0.smithi105.stdout:1313: writing smithi10548823-12 from 1577796 to 2033680 tid 2 2024-04-20T02:51:54.494 INFO:tasks.rados.rados.0.smithi105.stdout:1307: done (4 left) 2024-04-20T02:51:54.494 INFO:tasks.rados.rados.0.smithi105.stdout:1314: read oid 30 snap -1 2024-04-20T02:51:54.494 INFO:tasks.rados.rados.0.smithi105.stdout:1314: expect (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:51:54.494 INFO:tasks.rados.rados.0.smithi105.stdout:1315: set_chunk oid 49 target oid 2024-04-20T02:51:54.508 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:54.508 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1315: set_chunk oid 49 offset: 177664 length: 766976 target oid fa7f29cd7183e8c6bc1c85c1277f390f60c93dab51d3d5f61be94305d21cf8bc offset: 0 snap -1 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1316: tier_flush oid 41 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1316: tier_flush oid 41 snap -1 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1306: finishing write tid 6 to smithi10548823-18 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1306: oid 18 updating version 0 to 91 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1306: oid 18 updating version 91 to 93 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1306: oid 18 version 93 is already newer than 89 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1306: oid 18 version 93 is already newer than 90 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1306: oid 18 version 93 is already newer than 92 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 93 (ObjNum 237 snap 22 seq_num 237) dirty exists 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1306: left oid 18 (ObjNum 237 snap 22 seq_num 237) 2024-04-20T02:51:54.519 INFO:tasks.rados.rados.0.smithi105.stdout:1306: done (6 left) 2024-04-20T02:51:54.520 INFO:tasks.rados.rados.0.smithi105.stdout:1317: set_chunk oid 31 target oid 2024-04-20T02:51:54.521 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:54.565 INFO:tasks.rados.rados.0.smithi105.stdout:1317: set_chunk oid 31 offset: 146432 length: 1809408 target oid d408c73da5b7c094397f0c895cf65d7e65431dfdd7ba96e881c1ed81927f4b37 offset: 0 snap -1 2024-04-20T02:51:54.566 INFO:tasks.rados.rados.0.smithi105.stdout:1318: write oid 3 current snap is 22 2024-04-20T02:51:54.566 INFO:tasks.rados.rados.0.smithi105.stdout:1318: seq_num 241 ranges {450487=541952,1636278=414209} 2024-04-20T02:51:54.570 INFO:tasks.rados.rados.0.smithi105.stdout:1318: writing smithi10548823-3 from 450487 to 992439 tid 1 2024-04-20T02:51:54.576 INFO:tasks.rados.rados.0.smithi105.stdout:1318: writing smithi10548823-3 from 1636278 to 2050487 tid 2 2024-04-20T02:51:54.576 INFO:tasks.rados.rados.0.smithi105.stdout:1319: read oid 9 snap -1 2024-04-20T02:51:54.576 INFO:tasks.rados.rados.0.smithi105.stdout:1319: expect (ObjNum 228 snap 22 seq_num 228) 2024-04-20T02:51:54.576 INFO:tasks.rados.rados.0.smithi105.stdout:1320: set_chunk oid 24 target oid 2024-04-20T02:51:54.606 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:54.607+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:54.689 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:54.690+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:54.690 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:54.690 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:54.700 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:54.790 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:54.790+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:54.829 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:54.829 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:54.839 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:54.924 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:54.924+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:54.993 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:54.993+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:55.007 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:55.007 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:55.017 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:55.093 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:55.094+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:55.186 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:55.186 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:55.194 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:55.194+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:55.195 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:55.281 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:51:55.295 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:55.296+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:55.326 INFO:tasks.rados.rados.0.smithi105.stdout:1320: set_chunk oid 24 offset: 83456 length: 282112 target oid e8e529957fda9fc2c271d3fed6fe744bb62b3f5d3f47db0b6e45afdd7c9fd9fc offset: 0 snap -1 2024-04-20T02:51:55.326 INFO:tasks.rados.rados.0.smithi105.stdout:1321: read oid 29 snap 21 2024-04-20T02:51:55.326 INFO:tasks.rados.rados.0.smithi105.stdout:1321: expect (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:51:55.327 INFO:tasks.rados.rados.0.smithi105.stdout:1322: set_chunk oid 47 target oid 2024-04-20T02:51:55.352 INFO:tasks.rados.rados.0.smithi105.stdout:1322: set_chunk oid 47 offset: 470016 length: 1647616 target oid 132cce32a72892c2b4be04801d657ca57d3894cb1d8f6b90d5da4c2e7500f55b offset: 0 snap -1 2024-04-20T02:51:55.352 INFO:tasks.rados.rados.0.smithi105.stdout:1323: read oid 32 snap -1 2024-04-20T02:51:55.352 INFO:tasks.rados.rados.0.smithi105.stdout:1323: expect (ObjNum 183 snap 17 seq_num 183) 2024-04-20T02:51:55.352 INFO:tasks.rados.rados.0.smithi105.stdout:1324: write oid 36 current snap is 22 2024-04-20T02:51:55.352 INFO:tasks.rados.rados.0.smithi105.stdout:1312: expect (ObjNum 88 snap 2 seq_num 88) 2024-04-20T02:51:55.371 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:51:55.393 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:55.393 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:55.396 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:55.396+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:55.404 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:55.471 INFO:tasks.rados.rados.0.smithi105.stdout:1316: got (2) No such file or directory 2024-04-20T02:51:55.471 INFO:tasks.rados.rados.0.smithi105.stdout:1316: got expected ENOENT (src dne) 2024-04-20T02:51:55.471 INFO:tasks.rados.rados.0.smithi105.stdout:1315: finishing set_chunk to oid 49 2024-04-20T02:51:55.471 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 96 (ObjNum 152 snap 10 seq_num 152) dirty exists 2024-04-20T02:51:55.471 INFO:tasks.rados.rados.0.smithi105.stdout:1317: finishing set_chunk to oid 31 2024-04-20T02:51:55.471 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 159 (ObjNum 238 snap 22 seq_num 238) dirty exists 2024-04-20T02:51:55.471 INFO:tasks.rados.rados.0.smithi105.stdout:1314: expect (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:51:55.497 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:55.498+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:55.522 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:55.541 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:55.560 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:55.579 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:55.580 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:55.580 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:55.580 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:55.581 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:55.581 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:55.591 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:55.599 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:55.600 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:55.600+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:55.619 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:55.640 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:55.660 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:55.661 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:55.661 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:55.661 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:55.661 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:55.661 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:55.661 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:55.661 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:55.662 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:55.714 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:55.714+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:55.732 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:55.732 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:55.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.146576135999993, 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.733 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:51:55.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:51:55.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:55.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:51:55.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.130557858000003, 2024-04-20T02:51:55.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:55.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:51:55.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:51:55.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:51:55.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:51:55.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:51:55.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:51:55.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:51:55.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:51:55.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.130404628999997, 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:51:55.748 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:55.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:51:55.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:51:55.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:51:55.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:51:55.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:51:55.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:51:55.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:55.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:51:55.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:51:55.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:51:55.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:51:55.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:51:55.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:51:55.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.130294620000001, 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.756 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:51:55.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:51:55.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:51:55.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:51:55.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:51:55.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:51:55.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:51:55.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:51:55.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:51:55.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:51:55.763 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.763 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.130264244000003, 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.764 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:55.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:51:55.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:55.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:51:55.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:51:55.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:51:55.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:51:55.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:51:55.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:51:55.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:51:55.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:51:55.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.125846824999996, 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.771 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:51:55.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:51:55.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:51:55.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:51:55.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:51:55.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:51:55.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:51:55.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:51:55.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:51:55.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:55.778 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.116595074000003, 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:51:55.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:51:55.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:51:55.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:51:55.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:51:55.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:51:55.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.114072930999995, 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.786 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.787 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.787 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.787 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:51:55.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.787 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:55.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:51:55.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:51:55.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:55.793 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.113988063999997, 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:51:55.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:51:55.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:51:55.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.113970453999997, 2024-04-20T02:51:55.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:51:55.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:51:55.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:51:55.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:51:55.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.764603846, 2024-04-20T02:51:55.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058137664999999998, 2024-04-20T02:51:55.807 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.807 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.807 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:55.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967112+0000", 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981895+0000", 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014783522 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981906+0000", 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0560000000000001e-05 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981912+0000", 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:51:55.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981956+0000", 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3455000000000003e-05 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981979+0000", 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3475000000000001e-05 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025222+0000", 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043242622000000001 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025250+0000", 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8520000000000001e-05 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 61.319595858, 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:51:55.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:51:55.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:51:55.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:51:55.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:51:55.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.816 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 61.313591070000001, 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:55.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.821 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.822 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:55.822 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:55.822 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.590374947000001, 2024-04-20T02:51:55.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056137746000000002, 2024-04-20T02:51:55.822 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.822 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.822 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141340+0000", 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142751+0000", 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014113540000000001 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142770+0000", 2024-04-20T02:51:55.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8450000000000001e-05 2024-04-20T02:51:55.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142780+0000", 2024-04-20T02:51:55.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0284e-05 2024-04-20T02:51:55.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142982+0000", 2024-04-20T02:51:55.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000202244 2024-04-20T02:51:55.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.143007+0000", 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4929000000000001e-05 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197446+0000", 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054438519999999997 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197479+0000", 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3597999999999999e-05 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2490 4.9 e90/81 4:9c87d36d:::smithi10548823-24:head v 90'246, mlcod=88'242)", 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.4872760239999998, 2024-04-20T02:51:55.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055338261999999999, 2024-04-20T02:51:55.829 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.829 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.829 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:51:55.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244440+0000", 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245684+0000", 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001243755 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245696+0000", 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.171e-05 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245706+0000", 2024-04-20T02:51:55.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0236e-05 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245756+0000", 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0571000000000001e-05 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245784+0000", 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7841999999999999e-05 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.299737+0000", 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053952420000000001 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.299779+0000", 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2014000000000001e-05 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.833 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2654 4.7 4:e9876a63:::smithi10548823-30:head [write 2075401~451067] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:51:55.834 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:51:55.834 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.2608428529999998, 2024-04-20T02:51:55.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063177293999999995, 2024-04-20T02:51:55.834 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.834 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.834 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.834 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.834 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.834 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2654 2024-04-20T02:51:55.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.835 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:51:55.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:51:55.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470871+0000", 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471109+0000", 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023773800000000001 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471129+0000", 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0100999999999999e-05 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471139+0000", 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0427e-05 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.493743+0000", 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02260334 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.493897+0000", 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015392800000000001 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:51:55.838 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.494161+0000", 2024-04-20T02:51:55.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026414899999999998 2024-04-20T02:51:55.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510453+0000", 2024-04-20T02:51:55.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016292054 2024-04-20T02:51:55.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510509+0000", 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6709000000000002e-05 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533920+0000", 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023410713999999999 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533989+0000", 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8421000000000001e-05 2024-04-20T02:51:55.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534051+0000", 2024-04-20T02:51:55.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2398999999999997e-05 2024-04-20T02:51:55.841 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.841 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.841 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2655 4.7 4:e9876a63:::smithi10548823-30:head [write 3107008~648769] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.2529398220000001, 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055363825999999998, 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2655 2024-04-20T02:51:55.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478775+0000", 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478959+0000", 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018453600000000001 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478981+0000", 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1523e-05 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478991+0000", 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0288e-05 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510544+0000", 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031552795000000002 2024-04-20T02:51:55.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510667+0000", 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012335699999999999 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510928+0000", 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000260487 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533864+0000", 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022936926 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533908+0000", 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3498000000000003e-05 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534060+0000", 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015223999999999999 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534113+0000", 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2612999999999999e-05 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534141+0000", 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7696e-05 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.848 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2656 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (57) in=7b,truncate 3798382] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.2527097449999998, 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055660077000000002, 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.849 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2656 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479006+0000", 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479011+0000", 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.578e-06 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479015+0000", 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3869999999999997e-06 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479019+0000", 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.703e-06 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533236+0000", 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054217551000000003 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533355+0000", 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011906899999999999 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533595+0000", 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023979599999999999 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534575+0000", 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00097944500000000001 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534599+0000", 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4371000000000001e-05 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534605+0000", 2024-04-20T02:51:55.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534631+0000", 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6469999999999999e-05 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534667+0000", 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5822999999999999e-05 2024-04-20T02:51:55.855 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.856 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.856 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2657 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:51:55.856 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:51:55.856 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.2526944459999996, 2024-04-20T02:51:55.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055802286, 2024-04-20T02:51:55.856 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.856 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:55.857 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.857 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.857 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.857 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2657 2024-04-20T02:51:55.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.857 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:51:55.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.858 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:51:55.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.858 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:51:55.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:55.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479024+0000", 2024-04-20T02:51:55.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.779e-06 2024-04-20T02:51:55.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479027+0000", 2024-04-20T02:51:55.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4529999999999998e-06 2024-04-20T02:51:55.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479030+0000", 2024-04-20T02:51:55.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.993e-06 2024-04-20T02:51:55.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534184+0000", 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055154305000000001 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534295+0000", 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110665 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534694+0000", 2024-04-20T02:51:55.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039911500000000002 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534753+0000", 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8649e-05 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534824+0000", 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1589999999999997e-05 2024-04-20T02:51:55.862 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.863 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.863 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.863 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2658 4.7 4:e2c0d65b:::smithi10548823-46:head [read 561152~397312] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:51:55.863 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:51:55.863 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.2526842450000002, 2024-04-20T02:51:55.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055503328999999997, 2024-04-20T02:51:55.863 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:55.863 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2658 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:55.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479034+0000", 2024-04-20T02:51:55.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5600000000000001e-06 2024-04-20T02:51:55.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:55.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479036+0000", 2024-04-20T02:51:55.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6280000000000001e-06 2024-04-20T02:51:55.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479039+0000", 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5969999999999999e-06 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534331+0000", 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055292556 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534409+0000", 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7411999999999997e-05 2024-04-20T02:51:55.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:55.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:55.868 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:55.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534536+0000", 2024-04-20T02:51:55.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012682500000000001 2024-04-20T02:51:55.868 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.868 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.868 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.868 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:55.868 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:55.868 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:55.870 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:51:55.870 INFO:tasks.rados.rados.0.smithi105.stdout:1318: finishing write tid 1 to smithi10548823-3 2024-04-20T02:51:55.870 INFO:tasks.rados.rados.0.smithi105.stdout:1318: finishing write tid 2 to smithi10548823-3 2024-04-20T02:51:55.871 INFO:tasks.rados.rados.0.smithi105.stdout:1318: finishing write tid 3 to smithi10548823-3 2024-04-20T02:51:55.871 INFO:tasks.rados.rados.0.smithi105.stdout:1318: finishing write tid 4 to smithi10548823-3 2024-04-20T02:51:55.871 INFO:tasks.rados.rados.0.smithi105.stdout:1318: oid 3 updating version 0 to 277 2024-04-20T02:51:55.871 INFO:tasks.rados.rados.0.smithi105.stdout:1318: oid 3 version 277 is already newer than 276 2024-04-20T02:51:55.871 INFO:tasks.rados.rados.0.smithi105.stdout:1318: oid 3 version 277 is already newer than 275 2024-04-20T02:51:55.871 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 277 (ObjNum 241 snap 22 seq_num 241) dirty exists 2024-04-20T02:51:55.871 INFO:tasks.rados.rados.0.smithi105.stdout:1318: left oid 3 (ObjNum 241 snap 22 seq_num 241) 2024-04-20T02:51:55.871 INFO:tasks.rados.rados.0.smithi105.stdout:1321: expect (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:51:55.872 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:55.870+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:55.970 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:55.971+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:56.040 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:56.040 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:56.040 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:56.040 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:56.050 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:51:56.057 INFO:tasks.rados.rados.0.smithi105.stdout:1320: finishing set_chunk to oid 24 2024-04-20T02:51:56.057 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 273 (ObjNum 234 snap 22 seq_num 234) dirty exists 2024-04-20T02:51:56.057 INFO:tasks.rados.rados.0.smithi105.stdout:1322: finishing set_chunk to oid 47 2024-04-20T02:51:56.058 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 470016 length: 1647616 2024-04-20T02:51:56.058 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-20T02:51:56.058 INFO:tasks.rados.rados.0.smithi105.stdout:1324: seq_num 242 ranges {467294=617201,1578407=488887} 2024-04-20T02:51:56.063 INFO:tasks.rados.rados.0.smithi105.stdout:1324: writing smithi10548823-36 from 467294 to 1084495 tid 1 2024-04-20T02:51:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:1324: writing smithi10548823-36 from 1578407 to 2067294 tid 2 2024-04-20T02:51:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:1312: done (13 left) 2024-04-20T02:51:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:1314: done (12 left) 2024-04-20T02:51:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:1315: done (11 left) 2024-04-20T02:51:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:1316: done (10 left) 2024-04-20T02:51:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:1317: done (9 left) 2024-04-20T02:51:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:1318: done (8 left) 2024-04-20T02:51:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:1320: done (7 left) 2024-04-20T02:51:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:1321: done (6 left) 2024-04-20T02:51:56.070 INFO:tasks.rados.rados.0.smithi105.stdout:1322: done (5 left) 2024-04-20T02:51:56.071 INFO:tasks.rados.rados.0.smithi105.stdout:1325: read oid 48 snap 21 2024-04-20T02:51:56.071 INFO:tasks.rados.rados.0.smithi105.stdout:1325: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:51:56.071 INFO:tasks.rados.rados.0.smithi105.stdout:1326: read oid 6 snap 21 2024-04-20T02:51:56.071 INFO:tasks.rados.rados.0.smithi105.stdout:1326: expect (ObjNum 194 snap 19 seq_num 194) 2024-04-20T02:51:56.071 INFO:tasks.rados.rados.0.smithi105.stdout:1327: set_chunk oid 2 target oid 2024-04-20T02:51:56.076 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:56.077+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:56.126 INFO:tasks.rados.rados.0.smithi105.stdout:1327: set_chunk oid 2 offset: 284672 length: 1904128 target oid 27b240b05ce75114b683ef257328c902fdd5cc3998dce9a6304f8777ca278278 offset: 0 snap -1 2024-04-20T02:51:56.127 INFO:tasks.rados.rados.0.smithi105.stdout:1328: read oid 20 snap 21 2024-04-20T02:51:56.127 INFO:tasks.rados.rados.0.smithi105.stdout:1328: expect (ObjNum 193 snap 19 seq_num 193) 2024-04-20T02:51:56.127 INFO:tasks.rados.rados.0.smithi105.stdout:1329: write (excl) oid 8 current snap is 22 2024-04-20T02:51:56.127 INFO:tasks.rados.rados.0.smithi105.stdout:1329: seq_num 243 ranges {484101=692450,1636889=447212} 2024-04-20T02:51:56.133 INFO:tasks.rados.rados.0.smithi105.stdout:1329: writing smithi10548823-8 from 484101 to 1176551 tid 1 2024-04-20T02:51:56.139 INFO:tasks.rados.rados.0.smithi105.stdout:1329: writing smithi10548823-8 from 1636889 to 2084101 tid 2 2024-04-20T02:51:56.140 INFO:tasks.rados.rados.0.smithi105.stdout:1325: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:51:56.173 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:56.173+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:56.218 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:56.218 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:56.218 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:56.218 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:56.218 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:56.229 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:51:56.275 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:56.276+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:56.374 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:56.374+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:56.391 INFO:tasks.rados.rados.0.smithi105.stdout:1326: expect (ObjNum 194 snap 19 seq_num 194) 2024-04-20T02:51:56.420 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:56.420 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:56.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.474846240999995, 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041399200999999997, 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945227+0000", 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947429+0000", 2024-04-20T02:51:56.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022028989999999999 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947448+0000", 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.821e-05 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947460+0000", 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2373e-05 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947511+0000", 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0844999999999999e-05 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947531+0000", 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0017000000000002e-05 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986580+0000", 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039049213999999999 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986627+0000", 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7070000000000002e-05 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1145 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'30, mlcod=34'26)", 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.456115542000006, 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026752594000000001, 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:56.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:51:56.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963958+0000", 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964662+0000", 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070467299999999995 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964678+0000", 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5597999999999999e-05 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964687+0000", 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2979999999999997e-06 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968184+0000", 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034972250000000001 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968211+0000", 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6829999999999999e-05 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990640+0000", 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022429317000000001 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990711+0000", 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0794e-05 2024-04-20T02:51:56.429 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:129 3.0 e42/21 3:0256f537:::3f82a762bf2204de901e10565f70e759ee2a26bf55ffef4c8d5027a47e96ec31:head v 42'78, mlcod=42'76)", 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.285426966999999, 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024418002000000001, 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:56.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:51:56.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134646+0000", 2024-04-20T02:51:56.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:56.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012265099999999999 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134789+0000", 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9924000000000001e-05 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134799+0000", 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0042e-05 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134842+0000", 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3791000000000001e-05 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134866+0000", 2024-04-20T02:51:56.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3791e-05 2024-04-20T02:51:56.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159017+0000", 2024-04-20T02:51:56.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024150919 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159065+0000", 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201000000000001e-05 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:134 3.0 e42/21 3:33aaac3f:::d0aca4dab23a9f22fb58211e350c7c1c1b7601bafbb496b93afbbc91e2ed921c:head v 42'81, mlcod=42'76)", 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.275404768000001, 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024591152000000002, 2024-04-20T02:51:56.436 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144949+0000", 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028058500000000002 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144959+0000", 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0335e-05 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144967+0000", 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6040000000000004e-06 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151637+0000", 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066694670000000001 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151659+0000", 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2855000000000001e-05 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169224+0000", 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017565041 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169261+0000", 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6179e-05 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1462 3.0 e44/21 3:29b349f1:::ea1112aea5e7be379214f0429a6e898cf4f4a05bdbb29b6cd15764b4776418b1:head v 44'85, mlcod=42'83)", 2024-04-20T02:51:56.440 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.242284883000004, 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034060275000000001, 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178932+0000", 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011426069999999999 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178944+0000", 2024-04-20T02:51:56.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.287e-05 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178962+0000", 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7669999999999999e-05 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179013+0000", 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1295000000000003e-05 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179038+0000", 2024-04-20T02:51:56.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4859999999999999e-05 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211813+0000", 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032775190000000003 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211850+0000", 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6216e-05 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1629 3.0 e51/21 3:088cc2f3:::a1744cc895df5e9d3e624b281fcb9edc57f3f342d1f108d4c38dc778f219a41a:head v 51'93, mlcod=45'91)", 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.616985546000002, 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035323188999999998, 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:56.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803088+0000", 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804187+0000", 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00109891 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804201+0000", 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3772e-05 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804213+0000", 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2335e-05 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804285+0000", 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1271000000000003e-05 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804306+0000", 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1359999999999999e-05 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838377+0000", 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034070994 2024-04-20T02:51:56.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838412+0000", 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4790999999999999e-05 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1653 4.6 e51/22 4:697da67d:::smithi10548823-48:head v 51'41, mlcod=45'37)", 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.995246438000002, 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037653640000000002, 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.449 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424827+0000", 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425066+0000", 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023957399999999999 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425117+0000", 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1082999999999999e-05 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425146+0000", 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8129000000000001e-05 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425217+0000", 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1402000000000005e-05 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425278+0000", 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1498e-05 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462456+0000", 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037177106000000001 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462481+0000", 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5851e-05 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1786 4.d e56/22 4:b376c20e:::smithi10548823-49:head v 56'80, mlcod=55'77)", 2024-04-20T02:51:56.453 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.608154900999999, 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028136330000000001, 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812644+0000", 2024-04-20T02:51:56.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072510100000000002 2024-04-20T02:51:56.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812656+0000", 2024-04-20T02:51:56.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1688999999999999e-05 2024-04-20T02:51:56.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812663+0000", 2024-04-20T02:51:56.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812707+0000", 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4573999999999997e-05 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812722+0000", 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5319e-05 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840021+0000", 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027298111999999999 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840056+0000", 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5086000000000003e-05 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.458 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1799 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'53, mlcod=54'49)", 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.34582588, 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050527214000000001, 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074614+0000", 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036631499999999998 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074624+0000", 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8330000000000006e-06 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074632+0000", 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3599999999999998e-06 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074683+0000", 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1137999999999997e-05 2024-04-20T02:51:56.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074704+0000", 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1616000000000001e-05 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124714+0000", 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050009642 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124776+0000", 2024-04-20T02:51:56.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1581000000000006e-05 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.331329783999998, 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046184680999999998, 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088743+0000", 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090327+0000", 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001584071 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090343+0000", 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5193e-05 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090354+0000", 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1336e-05 2024-04-20T02:51:56.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090411+0000", 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6589999999999999e-05 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090431+0000", 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0579000000000001e-05 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134883+0000", 2024-04-20T02:51:56.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044451877000000001 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134929+0000", 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6131000000000003e-05 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1800 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'54, mlcod=54'49)", 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.31892371, 2024-04-20T02:51:56.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027029681999999999, 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101150+0000", 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101284+0000", 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000134688 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101295+0000", 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:51:56.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101305+0000", 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0930999999999999e-05 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101347+0000", 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1522e-05 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101369+0000", 2024-04-20T02:51:56.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2027000000000001e-05 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128138+0000", 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026768626 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128180+0000", 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2568999999999998e-05 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.471 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1807 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'87, mlcod=56'82)", 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.311242884000002, 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026347280000000001, 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108830+0000", 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109221+0000", 2024-04-20T02:51:56.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039087699999999998 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109239+0000", 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8904000000000001e-05 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109271+0000", 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1319000000000003e-05 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120423+0000", 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011151912999999999 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120454+0000", 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1464999999999998e-05 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135158+0000", 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014703554000000001 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135179+0000", 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0965999999999999e-05 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2038 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'159, mlcod=0'0)", 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 31.838454311, 2024-04-20T02:51:56.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044306238999999997, 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:51:56.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:56.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581964+0000", 2024-04-20T02:51:56.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034420099999999998 2024-04-20T02:51:56.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581980+0000", 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6578999999999998e-05 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581991+0000", 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0949e-05 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582734+0000", 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074277299999999996 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582761+0000", 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6627000000000001e-05 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625882+0000", 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043121617000000001 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625926+0000", 2024-04-20T02:51:56.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3838999999999998e-05 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2039 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'160, mlcod=0'0)", 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 31.818056225999999, 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027318719000000002, 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:56.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602017+0000", 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602364+0000", 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000347139 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602375+0000", 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.128e-05 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602384+0000", 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2600000000000005e-06 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605223+0000", 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0028393400000000001 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605245+0000", 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2078000000000001e-05 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629297+0000", 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024051605 2024-04-20T02:51:56.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629337+0000", 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0126000000000002e-05 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2137 3.0 e70/21 3:0d26903d:::7146376437549e0b99264d159bf1e9c8bc43ac1c7eb519f65899cfd592a43a4a:head v 70'98, mlcod=55'96)", 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.635928241999999, 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024394852000000002, 2024-04-20T02:51:56.485 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784144+0000", 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785280+0000", 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001135741 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785299+0000", 2024-04-20T02:51:56.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8845999999999999e-05 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785310+0000", 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1450999999999999e-05 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785362+0000", 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1798000000000001e-05 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785388+0000", 2024-04-20T02:51:56.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5763999999999999e-05 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808507+0000", 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023118742000000001 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808541+0000", 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4258000000000003e-05 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.489 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2142 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'65, mlcod=66'61)", 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 28.603228828999999, 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033365672999999998, 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:56.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816981+0000", 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135952 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816994+0000", 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3405e-05 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817003+0000", 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4319999999999999e-06 2024-04-20T02:51:56.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817055+0000", 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1842999999999998e-05 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817076+0000", 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1375000000000001e-05 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850179+0000", 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033103051000000001 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850211+0000", 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1896000000000001e-05 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:51:56.494 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.278694214000001, 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039421115999999999, 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141379+0000", 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142797+0000", 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001418702 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142816+0000", 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8907999999999999e-05 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142825+0000", 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1020000000000003e-06 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142872+0000", 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6168999999999998e-05 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142891+0000", 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9561000000000001e-05 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180704+0000", 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037812643 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180801+0000", 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7361999999999994e-05 2024-04-20T02:51:56.498 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2280 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'105, mlcod=73'100)", 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.258012937, 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029657706999999998, 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:56.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162060+0000", 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162455+0000", 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039569799999999999 2024-04-20T02:51:56.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162472+0000", 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6820000000000002e-05 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162485+0000", 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2388999999999999e-05 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163758+0000", 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001273906 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163784+0000", 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5352000000000001e-05 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191663+0000", 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027879583999999999 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191719+0000", 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640000000000003e-05 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.502 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2559 3.0 e91/21 3:0870df8d:::348e4a2a55868b09db2e6d173951b4676cfef7376c5a81f7a82bb8f7ccc4ada5:head v 91'132, mlcod=91'130)", 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 7.9512571760000004, 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032176705999999999, 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468815+0000", 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469831+0000", 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001015076 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469853+0000", 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2713000000000001e-05 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469865+0000", 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1904e-05 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469910+0000", 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5114999999999997e-05 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469934+0000", 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4133000000000001e-05 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.500951+0000", 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031016851000000002 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.500994+0000", 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2496000000000001e-05 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.506 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2788 4.14 e98/97 4:2e88ea27:::smithi10548823-12:head v 98'31, mlcod=0'0)", 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:56.144245+0000", 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.275829512, 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038004836, 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144245+0000", 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144245+0000", 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144244+0000", 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144489+0000", 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024499599999999998 2024-04-20T02:51:56.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144512+0000", 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2989e-05 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144521+0000", 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2769999999999994e-06 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144580+0000", 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9119000000000001e-05 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144612+0000", 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1962000000000003e-05 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.182209+0000", 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037597034000000001 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:56.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.182250+0000", 2024-04-20T02:51:56.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0376000000000001e-05 2024-04-20T02:51:56.511 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.511 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.511 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.511 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:56.511 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:56.511 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:56.513 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:51:56.513 INFO:tasks.rados.rados.0.smithi105.stdout:1324: finishing write tid 1 to smithi10548823-36 2024-04-20T02:51:56.513 INFO:tasks.rados.rados.0.smithi105.stdout:1324: finishing write tid 2 to smithi10548823-36 2024-04-20T02:51:56.513 INFO:tasks.rados.rados.0.smithi105.stdout:1324: finishing write tid 3 to smithi10548823-36 2024-04-20T02:51:56.513 INFO:tasks.rados.rados.0.smithi105.stdout:1324: finishing write tid 4 to smithi10548823-36 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:1324: oid 36 updating version 0 to 276 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:1324: oid 36 version 276 is already newer than 274 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:1324: oid 36 version 276 is already newer than 275 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 276 (ObjNum 242 snap 22 seq_num 242) dirty exists 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:1324: left oid 36 (ObjNum 242 snap 22 seq_num 242) 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:1324: done (9 left) 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:1325: done (8 left) 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:1326: done (7 left) 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:1330: write (excl) oid 36 current snap is 22 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:1327: finishing set_chunk to oid 2 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 284672 length: 1904128 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 155 snap 10 seq_num 155) dirty exists 2024-04-20T02:51:56.514 INFO:tasks.rados.rados.0.smithi105.stdout:1330: seq_num 244 ranges {500908=484052,1411724=689184} 2024-04-20T02:51:56.515 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:56.512+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:56.516 INFO:tasks.rados.rados.0.smithi105.stdout:1330: writing smithi10548823-36 from 500908 to 984960 tid 1 2024-04-20T02:51:56.523 INFO:tasks.rados.rados.0.smithi105.stdout:1330: writing smithi10548823-36 from 1411724 to 2100908 tid 2 2024-04-20T02:51:56.524 INFO:tasks.rados.rados.0.smithi105.stdout:1327: done (7 left) 2024-04-20T02:51:56.524 INFO:tasks.rados.rados.0.smithi105.stdout:1331: write (excl) oid 17 current snap is 22 2024-04-20T02:51:56.524 INFO:tasks.rados.rados.0.smithi105.stdout:1328: expect (ObjNum 193 snap 19 seq_num 193) 2024-04-20T02:51:56.614 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:56.614+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:56.648 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:56.648 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:56.648 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:51:56.648 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:56.658 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:51:56.692 INFO:tasks.rados.rados.0.smithi105.stdout:1319: expect (ObjNum 228 snap 22 seq_num 228) 2024-04-20T02:51:56.714 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:56.715+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:56.815 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:56.815+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:56.821 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:56.821 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:51:56.821 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:51:56.821 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:51:56.821 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:56.832 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:51:56.916 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:56.916+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:56.917 INFO:tasks.rados.rados.0.smithi105.stdout:1309: expect (ObjNum 222 snap 21 seq_num 222) 2024-04-20T02:51:57.011 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:51:57.011 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:51:57.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:51:57.011 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:51:57.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.011 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:57.011 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:57.011 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 70.466184526000006, 2024-04-20T02:51:57.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:51:57.011 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:51:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 70.459963451999997, 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:51:57.017 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:51:57.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 70.451563247999999, 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.023 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:51:57.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:51:57.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 70.451220602999996, 2024-04-20T02:51:57.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:51:57.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:51:57.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:51:57.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 62.644898599999998, 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:51:57.034 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:51:57.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 62.637635637999999, 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.040 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:51:57.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:51:57.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 62.631693108999997, 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:51:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:51:57.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 62.631490116999998, 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:51:57.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:51:57.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.056 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 62.631467190000002, 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:51:57.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:51:57.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:51:57.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 62.621383455999997, 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:51:57.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:51:57.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 62.604498047999996, 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:51:57.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:51:57.069 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 62.588221607000001, 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.070 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:51:57.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.074 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.563666134999998, 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.075 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:51:57.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:51:57.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.249350358999997, 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:57.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:51:57.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:51:57.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:51:57.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2545 4.9 e91/81 4:95e19a88:::smithi10548823-40:head v 91'253, mlcod=91'251)", 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.5947132439999994, 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054757404000000003, 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415210+0000", 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416004+0000", 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079381199999999997 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416022+0000", 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7969e-05 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416031+0000", 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9279999999999999e-06 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416100+0000", 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8891000000000002e-05 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416128+0000", 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8203000000000001e-05 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469912+0000", 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053783794000000003 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469968+0000", 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6474000000000002e-05 2024-04-20T02:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.6616844369999999, 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:51:57.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:51:57.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:51:57.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2700 4.0 4:007b2a87:::smithi10548823-39:head [write 1967200~714747] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.6531942429999997, 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061846287, 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2700 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356728+0000", 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356879+0000", 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000150958 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356894+0000", 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5132e-05 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356906+0000", 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2071e-05 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366666+0000", 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0097595289999999994 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366774+0000", 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108527 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366956+0000", 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181359 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390515+0000", 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023559902000000001 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406790+0000", 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016274555999999999 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418488+0000", 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011697591 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418529+0000", 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1545000000000003e-05 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418576+0000", 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6903000000000001e-05 2024-04-20T02:51:57.102 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2701 4.0 4:007b2a87:::smithi10548823-39:head [write 3421983~426820] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.6460182510000001, 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054799176999999998, 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.103 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2701 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363904+0000", 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364154+0000", 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000250179 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364171+0000", 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6748e-05 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364182+0000", 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1371e-05 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390552+0000", 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026369607 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390652+0000", 2024-04-20T02:51:57.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010059499999999999 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390857+0000", 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020445199999999999 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418224+0000", 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027367316999999999 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418282+0000", 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8279000000000002e-05 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418618+0000", 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000335967 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418661+0000", 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2614999999999998e-05 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418705+0000", 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3993999999999999e-05 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2702 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (57) in=7b,truncate 3848803] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.6457321970000001, 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058669165000000002, 2024-04-20T02:51:57.109 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2702 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364191+0000", 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364196+0000", 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4490000000000001e-06 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364201+0000", 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5040000000000002e-06 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364206+0000", 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.405895+0000", 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041689047 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406021+0000", 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000125561 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406541+0000", 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052000699999999996 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418451+0000", 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011910041999999999 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418747+0000", 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029622399999999999 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422704+0000", 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039571859999999997 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422781+0000", 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.661e-05 2024-04-20T02:51:57.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422861+0000", 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9987999999999994e-05 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2703 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.6457131450000002, 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058940278999999998, 2024-04-20T02:51:57.115 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2703 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364213+0000", 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4830000000000002e-06 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364220+0000", 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.007e-06 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364225+0000", 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0350000000000001e-06 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406820+0000", 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042595135999999999 2024-04-20T02:51:57.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406923+0000", 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102846 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422897+0000", 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015974202999999999 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423015+0000", 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117581 2024-04-20T02:51:57.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:51:57.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:51:57.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:51:57.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423151+0000", 2024-04-20T02:51:57.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013635899999999999 2024-04-20T02:51:57.120 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.120 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.120 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.120 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:51:57.120 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:51:57.120 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:51:57.122 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:51:57.122 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:57.121+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:57.146 INFO:tasks.rados.rados.0.smithi105.stdout:1323: expect (ObjNum 183 snap 17 seq_num 183) 2024-04-20T02:51:57.154 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:51:57.154 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:51:57.155 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-04-20T02:51:57.155 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-04-20T02:51:57.155 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:51:57.223 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:57.223+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:57.256 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:57.256 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:57.256 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:57.256 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:57.266 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:51:57.310 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:51:57.323 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:57.324+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:57.363 INFO:tasks.rados.rados.0.smithi105.stdout:1331: seq_num 245 ranges {517715=559301,1870206=247509} 2024-04-20T02:51:57.370 INFO:tasks.rados.rados.0.smithi105.stdout:1331: writing smithi10548823-17 from 517715 to 1077016 tid 1 2024-04-20T02:51:57.377 INFO:tasks.rados.rados.0.smithi105.stdout:1331: writing smithi10548823-17 from 1870206 to 2117715 tid 2 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1309: done (7 left) 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1319: done (6 left) 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1323: done (5 left) 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1328: done (4 left) 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1332: read oid 11 snap -1 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1332: expect (ObjNum 210 snap 19 seq_num 210) 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1333: write oid 22 current snap is 22 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1313: finishing write tid 1 to smithi10548823-12 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1313: finishing write tid 2 to smithi10548823-12 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1313: finishing write tid 3 to smithi10548823-12 2024-04-20T02:51:57.378 INFO:tasks.rados.rados.0.smithi105.stdout:1333: seq_num 246 ranges {534522=634550,1812335=322187} 2024-04-20T02:51:57.384 INFO:tasks.rados.rados.0.smithi105.stdout:1333: writing smithi10548823-22 from 534522 to 1169072 tid 1 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:1333: writing smithi10548823-22 from 1812335 to 2134522 tid 2 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:1334: write (excl) oid 27 current snap is 22 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:1313: finishing write tid 4 to smithi10548823-12 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:1313: oid 12 updating version 0 to 33 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:1313: oid 12 version 33 is already newer than 32 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:1313: oid 12 version 33 is already newer than 31 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 33 (ObjNum 240 snap 22 seq_num 240) dirty exists 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:1313: left oid 12 (ObjNum 240 snap 22 seq_num 240) 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:1330: finishing write tid 1 to smithi10548823-36 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:1330: finishing write tid 2 to smithi10548823-36 2024-04-20T02:51:57.391 INFO:tasks.rados.rados.0.smithi105.stdout:1334: seq_num 247 ranges {551329=709799,1870817=280512} 2024-04-20T02:51:57.395 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:57.395 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:57.395 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:51:57.395 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:51:57.396 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:57.399 INFO:tasks.rados.rados.0.smithi105.stdout:1334: writing smithi10548823-27 from 551329 to 1261128 tid 1 2024-04-20T02:51:57.405 INFO:tasks.rados.rados.0.smithi105.stdout:1334: writing smithi10548823-27 from 1870817 to 2151329 tid 2 2024-04-20T02:51:57.405 INFO:tasks.rados.rados.0.smithi105.stdout:1313: done (6 left) 2024-04-20T02:51:57.405 INFO:tasks.rados.rados.0.smithi105.stdout:1335: read oid 42 snap -1 2024-04-20T02:51:57.405 INFO:tasks.rados.rados.0.smithi105.stdout:1335: expect (ObjNum 222 snap 21 seq_num 222) 2024-04-20T02:51:57.405 INFO:tasks.rados.rados.0.smithi105.stdout:1336: set_chunk oid 15 target oid 2024-04-20T02:51:57.406 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:51:57.424 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:57.425+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:57.526 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:51:57.527 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:51:57.528 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:57.528+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:57.539 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-04-20T02:51:57.459127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":256892629,"num_objects":176,"num_object_clones":16,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":1,"num_read":1987,"num_read_kb":1632146,"num_write":1418,"num_write_kb":388760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":50387536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":33,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":745428,"kb_used_data":496684,"kb_used_omap":12,"kb_used_meta":245427,"kb_avail":754229292,"statfs":{"total":773094113280,"available":772330795008,"internally_reserved":0,"allocated":508604416,"data_stored":541513478,"data_compressed":11854713,"data_compressed_allocated":12574720,"data_compressed_original":29831168,"omap_allocated":12807,"internal_metadata":251317753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19159471,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":72,"num_read_kb":90010,"num_write":51,"num_write_kb":20146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001179"},"pg_stats":[{"pgid":"4.19","version":"95'273","reported_seq":582,"reported_epoch":96,"state":"unknown","last_fresh":"2024-04-20T02:51:54.283756+0000","last_change":"2024-04-20T02:51:54.283756+0000","last_active":"2024-04-20T02:51:54.283756+0000","last_peered":"2024-04-20T02:51:52.847901+0000","last_clean":"2024-04-20T02:51:52.847901+0000","last_became_active":"2024-04-20T02:51:39.132079+0000","last_became_peered":"2024-04-20T02:51:39.132079+0000","last_unstale":"2024-04-20T02:51:54.283756+0000","last_undegraded":"2024-04-20T02:51:54.283756+0000","last_fullsized":"2024-04-20T02:51:54.283756+0000","mapping_epoch":96,"log_start":"95'268","ondisk_log_start":"95'268","created":96,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":5,"last_scrub":"87'238","last_scrub_stamp":"2024-04-20T02:51:45.092632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:45.092632+0000","objects_scrubbed":9,"log_size":5,"log_dups_size":436,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00347503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13179195,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":18,"num_read_kb":21998,"num_write":14,"num_write_kb":4316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-20T02:51:54.255147+0000","last_change":"2024-04-20T02:51:54.255147+0000","last_active":"2024-04-20T02:51:54.255147+0000","last_peered":"2024-04-20T02:51:54.255147+0000","last_clean":"2024-04-20T02:51:54.255147+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T02:51:54.255147+0000","last_undegraded":"2024-04-20T02:51:54.255147+0000","last_fullsized":"2024-04-20T02:51:54.255147+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:54.255147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:51:54.255147+0000","last_clean_scrub_stamp":"2024-04-20T02:51:54.255147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-20T02:51:54.255147+0000","last_change":"2024-04-20T02:51:54.255147+0000","last_active":"2024-04-20T02:51:54.255147+0000","last_peered":"2024-04-20T02:51:54.255147+0000","last_clean":"2024-04-20T02:51:54.255147+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T02:51:54.255147+0000","last_undegraded":"2024-04-20T02:51:54.255147+0000","last_fullsized":"2024-04-20T02:51:54.255147+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:54.255147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:51:54.255147+0000","last_clean_scrub_stamp":"2024-04-20T02:51:54.255147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"95'107","reported_seq":393,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-20T02:51:54.484216+0000","last_change":"2024-04-20T02:51:48.242518+0000","last_active":"2024-04-20T02:51:54.484216+0000","last_peered":"2024-04-20T02:51:54.484216+0000","last_clean":"2024-04-20T02:51:54.484216+0000","last_became_active":"2024-04-20T02:51:18.508043+0000","last_became_peered":"2024-04-20T02:51:18.508043+0000","last_unstale":"2024-04-20T02:51:54.484216+0000","last_undegraded":"2024-04-20T02:51:54.484216+0000","last_fullsized":"2024-04-20T02:51:54.484216+0000","mapping_epoch":46,"log_start":"93'102","ondisk_log_start":"93'102","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'101","last_scrub_stamp":"2024-04-20T02:51:34.098939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:34.098939+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":98,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:56.647060+0000","scrub_duration":0.0019988520000000002,"objects_trimmed":0,"snaptrim_duration":0.00081858500000000004,"stat_sum":{"num_bytes":5325417,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":35292,"num_write":5,"num_write_kb":1804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.559359+0000","last_change":"2024-04-20T02:50:27.643272+0000","last_active":"2024-04-20T02:51:51.559359+0000","last_peered":"2024-04-20T02:51:51.559359+0000","last_clean":"2024-04-20T02:51:51.559359+0000","last_became_active":"2024-04-20T02:50:27.642845+0000","last_became_peered":"2024-04-20T02:50:27.642845+0000","last_unstale":"2024-04-20T02:51:51.559359+0000","last_undegraded":"2024-04-20T02:51:51.559359+0000","last_fullsized":"2024-04-20T02:51:51.559359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"95'103","reported_seq":373,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.422784+0000","last_change":"2024-04-20T02:51:48.241945+0000","last_active":"2024-04-20T02:51:52.422784+0000","last_peered":"2024-04-20T02:51:52.422784+0000","last_clean":"2024-04-20T02:51:52.422784+0000","last_became_active":"2024-04-20T02:51:18.505708+0000","last_became_peered":"2024-04-20T02:51:18.505708+0000","last_unstale":"2024-04-20T02:51:52.422784+0000","last_undegraded":"2024-04-20T02:51:52.422784+0000","last_fullsized":"2024-04-20T02:51:52.422784+0000","mapping_epoch":22,"log_start":"94'98","ondisk_log_start":"94'98","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":94,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.922550+0000","scrub_duration":0.001820667,"objects_trimmed":0,"snaptrim_duration":0.00068793399999999996,"stat_sum":{"num_bytes":13688837,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":45,"num_read_kb":38218,"num_write":23,"num_write_kb":7278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.266756+0000","last_change":"2024-04-20T02:50:27.638986+0000","last_active":"2024-04-20T02:51:51.266756+0000","last_peered":"2024-04-20T02:51:51.266756+0000","last_clean":"2024-04-20T02:51:51.266756+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:51:51.266756+0000","last_undegraded":"2024-04-20T02:51:51.266756+0000","last_fullsized":"2024-04-20T02:51:51.266756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":97,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.351043+0000","last_change":"2024-04-20T02:51:41.093997+0000","last_active":"2024-04-20T02:51:52.351043+0000","last_peered":"2024-04-20T02:51:52.351043+0000","last_clean":"2024-04-20T02:51:52.351043+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:51:52.351043+0000","last_undegraded":"2024-04-20T02:51:52.351043+0000","last_fullsized":"2024-04-20T02:51:52.351043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:41.093905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:41.093905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:01.045646+0000","scrub_duration":0.0019242129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"94'207","reported_seq":517,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.285532+0000","last_change":"2024-04-20T02:51:48.248187+0000","last_active":"2024-04-20T02:51:52.285532+0000","last_peered":"2024-04-20T02:51:52.285532+0000","last_clean":"2024-04-20T02:51:52.285532+0000","last_became_active":"2024-04-20T02:51:18.502004+0000","last_became_peered":"2024-04-20T02:51:18.502004+0000","last_unstale":"2024-04-20T02:51:52.285532+0000","last_undegraded":"2024-04-20T02:51:52.285532+0000","last_fullsized":"2024-04-20T02:51:52.285532+0000","mapping_epoch":47,"log_start":"91'204","ondisk_log_start":"91'204","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'157","last_scrub_stamp":"2024-04-20T02:51:19.377186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.377186+0000","objects_scrubbed":5,"log_size":3,"log_dups_size":200,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:38.408177+0000","scrub_duration":0.0026503899999999999,"objects_trimmed":0,"snaptrim_duration":0.0027527430000000002,"stat_sum":{"num_bytes":3680733,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":43,"num_read_kb":69678,"num_write":15,"num_write_kb":4725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"3.1","version":"94'162","reported_seq":268,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.285166+0000","last_change":"2024-04-20T02:50:34.239216+0000","last_active":"2024-04-20T02:51:52.285166+0000","last_peered":"2024-04-20T02:51:52.285166+0000","last_clean":"2024-04-20T02:51:52.285166+0000","last_became_active":"2024-04-20T02:50:34.238906+0000","last_became_peered":"2024-04-20T02:50:34.238906+0000","last_unstale":"2024-04-20T02:51:52.285166+0000","last_undegraded":"2024-04-20T02:51:52.285166+0000","last_fullsized":"2024-04-20T02:51:52.285166+0000","mapping_epoch":21,"log_start":"92'159","ondisk_log_start":"92'159","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":184,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.390119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25193854,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":150,"num_read_kb":6891,"num_write":190,"num_write_kb":57991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.559087+0000","last_change":"2024-04-20T02:51:46.280098+0000","last_active":"2024-04-20T02:51:51.559087+0000","last_peered":"2024-04-20T02:51:51.559087+0000","last_clean":"2024-04-20T02:51:51.559087+0000","last_became_active":"2024-04-20T02:51:46.279750+0000","last_became_peered":"2024-04-20T02:51:46.279750+0000","last_unstale":"2024-04-20T02:51:51.559087+0000","last_undegraded":"2024-04-20T02:51:51.559087+0000","last_fullsized":"2024-04-20T02:51:51.559087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:28.222203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:28.222203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:54.983534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"93'88","reported_seq":326,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.285812+0000","last_change":"2024-04-20T02:51:48.248213+0000","last_active":"2024-04-20T02:51:52.285812+0000","last_peered":"2024-04-20T02:51:52.285812+0000","last_clean":"2024-04-20T02:51:52.285812+0000","last_became_active":"2024-04-20T02:51:18.504889+0000","last_became_peered":"2024-04-20T02:51:18.504889+0000","last_unstale":"2024-04-20T02:51:52.285812+0000","last_undegraded":"2024-04-20T02:51:52.285812+0000","last_fullsized":"2024-04-20T02:51:52.285812+0000","mapping_epoch":22,"log_start":"88'83","ondisk_log_start":"88'83","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'57","last_scrub_stamp":"2024-04-20T02:51:21.386796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:21.386796+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":82,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:27.553181+0000","scrub_duration":0.0018878359999999999,"objects_trimmed":0,"snaptrim_duration":0.0027467189999999999,"stat_sum":{"num_bytes":5233380,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":11430,"num_write":11,"num_write_kb":2792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"3.2","version":"93'153","reported_seq":350,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.128616+0000","last_change":"2024-04-20T02:51:44.446457+0000","last_active":"2024-04-20T02:51:51.128616+0000","last_peered":"2024-04-20T02:51:51.128616+0000","last_clean":"2024-04-20T02:51:51.128616+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:51:51.128616+0000","last_undegraded":"2024-04-20T02:51:51.128616+0000","last_fullsized":"2024-04-20T02:51:51.128616+0000","mapping_epoch":49,"log_start":"93'150","ondisk_log_start":"93'150","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'130","last_scrub_stamp":"2024-04-20T02:51:44.446332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:51:44.446332+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":158,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:11.809200+0000","scrub_duration":0.0039636890000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25485736,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":133,"num_read_kb":8281,"num_write":177,"num_write_kb":65393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":250,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.366065+0000","last_change":"2024-04-20T02:51:42.076681+0000","last_active":"2024-04-20T02:51:52.366065+0000","last_peered":"2024-04-20T02:51:52.366065+0000","last_clean":"2024-04-20T02:51:52.366065+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:51:52.366065+0000","last_undegraded":"2024-04-20T02:51:52.366065+0000","last_fullsized":"2024-04-20T02:51:52.366065+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:51:42.076506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:51:42.076506+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:59.869381+0000","scrub_duration":0.002077785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.129937+0000","last_change":"2024-04-20T02:51:30.923927+0000","last_active":"2024-04-20T02:51:51.129937+0000","last_peered":"2024-04-20T02:51:51.129937+0000","last_clean":"2024-04-20T02:51:51.129937+0000","last_became_active":"2024-04-20T02:51:01.151544+0000","last_became_peered":"2024-04-20T02:51:01.151544+0000","last_unstale":"2024-04-20T02:51:51.129937+0000","last_undegraded":"2024-04-20T02:51:51.129937+0000","last_fullsized":"2024-04-20T02:51:51.129937+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:30.923833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:30.923833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:47.551592+0000","scrub_duration":0.001311373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"56'33","reported_seq":250,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.559386+0000","last_change":"2024-04-20T02:51:48.254392+0000","last_active":"2024-04-20T02:51:51.559386+0000","last_peered":"2024-04-20T02:51:51.559386+0000","last_clean":"2024-04-20T02:51:51.559386+0000","last_became_active":"2024-04-20T02:51:18.502179+0000","last_became_peered":"2024-04-20T02:51:18.502179+0000","last_unstale":"2024-04-20T02:51:51.559386+0000","last_undegraded":"2024-04-20T02:51:51.559386+0000","last_fullsized":"2024-04-20T02:51:51.559386+0000","mapping_epoch":22,"log_start":"53'30","ondisk_log_start":"53'30","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'33","last_scrub_stamp":"2024-04-20T02:51:32.144366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.144366+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:49.222704+0000","scrub_duration":0.01361505,"objects_trimmed":0,"snaptrim_duration":0.00064738000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"3.0","version":"95'142","reported_seq":233,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.399389+0000","last_change":"2024-04-20T02:50:34.241909+0000","last_active":"2024-04-20T02:51:52.399389+0000","last_peered":"2024-04-20T02:51:52.399389+0000","last_clean":"2024-04-20T02:51:52.399389+0000","last_became_active":"2024-04-20T02:50:34.241613+0000","last_became_peered":"2024-04-20T02:50:34.241613+0000","last_unstale":"2024-04-20T02:51:52.399389+0000","last_undegraded":"2024-04-20T02:51:52.399389+0000","last_fullsized":"2024-04-20T02:51:52.399389+0000","mapping_epoch":21,"log_start":"95'139","ondisk_log_start":"95'139","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":151,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:54.710965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22518794,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":118,"num_read_kb":3314,"num_write":173,"num_write_kb":53521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":95,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.559041+0000","last_change":"2024-04-20T02:51:41.175139+0000","last_active":"2024-04-20T02:51:51.559041+0000","last_peered":"2024-04-20T02:51:51.559041+0000","last_clean":"2024-04-20T02:51:51.559041+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:51:51.559041+0000","last_undegraded":"2024-04-20T02:51:51.559041+0000","last_fullsized":"2024-04-20T02:51:51.559041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:41.175035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:41.175035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:46.711136+0000","scrub_duration":0.0014447850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"95'83","reported_seq":343,"reported_epoch":95,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:51:52.454280+0000","last_change":"2024-04-20T02:51:51.262054+0000","last_active":"2024-04-20T02:51:52.454280+0000","last_peered":"2024-04-20T02:51:52.454280+0000","last_clean":"2024-04-20T02:51:52.454280+0000","last_became_active":"2024-04-20T02:51:47.240817+0000","last_became_peered":"2024-04-20T02:51:47.240817+0000","last_unstale":"2024-04-20T02:51:52.454280+0000","last_undegraded":"2024-04-20T02:51:52.454280+0000","last_fullsized":"2024-04-20T02:51:52.454280+0000","mapping_epoch":46,"log_start":"94'78","ondisk_log_start":"94'78","created":22,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'50","last_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.487770+0000","objects_scrubbed":4,"log_size":5,"log_dups_size":131,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019136573,"objects_trimmed":0,"snaptrim_duration":0.93387975999999995,"stat_sum":{"num_bytes":12750861,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":91496,"num_write":42,"num_write_kb":12838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2757906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"3.3","version":"96'136","reported_seq":233,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-20T02:51:54.566156+0000","last_change":"2024-04-20T02:50:34.242061+0000","last_active":"2024-04-20T02:51:54.566156+0000","last_peered":"2024-04-20T02:51:54.566156+0000","last_clean":"2024-04-20T02:51:54.566156+0000","last_became_active":"2024-04-20T02:50:34.241638+0000","last_became_peered":"2024-04-20T02:50:34.241638+0000","last_unstale":"2024-04-20T02:51:54.566156+0000","last_undegraded":"2024-04-20T02:51:54.566156+0000","last_fullsized":"2024-04-20T02:51:54.566156+0000","mapping_epoch":21,"log_start":"96'133","ondisk_log_start":"96'133","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":139,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:58.050023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31206787,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":119,"num_read_kb":4911,"num_write":164,"num_write_kb":50974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.777249+0000","last_change":"2024-04-20T02:51:52.777359+0000","last_active":"2024-04-20T02:51:52.777249+0000","last_peered":"2024-04-20T02:51:52.777249+0000","last_clean":"2024-04-20T02:51:52.777249+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:51:52.777249+0000","last_undegraded":"2024-04-20T02:51:52.777249+0000","last_fullsized":"2024-04-20T02:51:52.777249+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:52.777234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:52.777234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:10.584062+0000","scrub_duration":0.0012962270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"56'17","reported_seq":224,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.266653+0000","last_change":"2024-04-20T02:51:48.246301+0000","last_active":"2024-04-20T02:51:51.266653+0000","last_peered":"2024-04-20T02:51:51.266653+0000","last_clean":"2024-04-20T02:51:51.266653+0000","last_became_active":"2024-04-20T02:51:18.509276+0000","last_became_peered":"2024-04-20T02:51:18.509276+0000","last_unstale":"2024-04-20T02:51:51.266653+0000","last_undegraded":"2024-04-20T02:51:51.266653+0000","last_fullsized":"2024-04-20T02:51:51.266653+0000","mapping_epoch":22,"log_start":"53'14","ondisk_log_start":"53'14","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:51:22.420320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:22.420320+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.257627+0000","scrub_duration":0.0016858400000000001,"objects_trimmed":0,"snaptrim_duration":0.00179882,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.2","version":"0'0","reported_seq":209,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.266690+0000","last_change":"2024-04-20T02:51:48.246080+0000","last_active":"2024-04-20T02:51:51.266690+0000","last_peered":"2024-04-20T02:51:51.266690+0000","last_clean":"2024-04-20T02:51:51.266690+0000","last_became_active":"2024-04-20T02:51:18.506462+0000","last_became_peered":"2024-04-20T02:51:18.506462+0000","last_unstale":"2024-04-20T02:51:51.266690+0000","last_undegraded":"2024-04-20T02:51:51.266690+0000","last_fullsized":"2024-04-20T02:51:51.266690+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:33.440257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:51:33.440257+0000","last_clean_scrub_stamp":"2024-04-20T02:51:33.440257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:00.147584+0000","scrub_duration":0.001747176,"objects_trimmed":0,"snaptrim_duration":0.0016174270000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.266556+0000","last_change":"2024-04-20T02:50:28.091519+0000","last_active":"2024-04-20T02:51:51.266556+0000","last_peered":"2024-04-20T02:51:51.266556+0000","last_clean":"2024-04-20T02:51:51.266556+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:51:51.266556+0000","last_undegraded":"2024-04-20T02:51:51.266556+0000","last_fullsized":"2024-04-20T02:51:51.266556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"95'72","reported_seq":314,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.424397+0000","last_change":"2024-04-20T02:51:48.274584+0000","last_active":"2024-04-20T02:51:52.424397+0000","last_peered":"2024-04-20T02:51:52.424397+0000","last_clean":"2024-04-20T02:51:52.424397+0000","last_became_active":"2024-04-20T02:51:43.136495+0000","last_became_peered":"2024-04-20T02:51:43.136495+0000","last_unstale":"2024-04-20T02:51:52.424397+0000","last_undegraded":"2024-04-20T02:51:52.424397+0000","last_fullsized":"2024-04-20T02:51:52.424397+0000","mapping_epoch":62,"log_start":"91'69","ondisk_log_start":"91'69","created":22,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'47","last_scrub_stamp":"2024-04-20T02:51:28.917651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:28.917651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":113,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001846091,"objects_trimmed":0,"snaptrim_duration":0.00043833000000000001,"stat_sum":{"num_bytes":4759965,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":146,"num_read_kb":192942,"num_write":30,"num_write_kb":8039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8110636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.9","version":"95'273","reported_seq":582,"reported_epoch":96,"state":"peering","last_fresh":"2024-04-20T02:51:54.283756+0000","last_change":"2024-04-20T02:51:54.285293+0000","last_active":"2024-04-20T02:51:54.283756+0000","last_peered":"2024-04-20T02:51:52.847901+0000","last_clean":"2024-04-20T02:51:52.847901+0000","last_became_active":"2024-04-20T02:51:39.132079+0000","last_became_peered":"2024-04-20T02:51:39.132079+0000","last_unstale":"2024-04-20T02:51:54.283756+0000","last_undegraded":"2024-04-20T02:51:54.283756+0000","last_fullsized":"2024-04-20T02:51:54.283756+0000","mapping_epoch":22,"log_start":"95'268","ondisk_log_start":"95'268","created":22,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'238","last_scrub_stamp":"2024-04-20T02:51:45.092632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:45.092632+0000","objects_scrubbed":9,"log_size":5,"log_dups_size":436,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:49.797308+0000","scrub_duration":0.00347503,"objects_trimmed":1,"snaptrim_duration":0.0052990659999999998,"stat_sum":{"num_bytes":13179195,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":21997,"num_write":13,"num_write_kb":4316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.a","version":"90'53","reported_seq":270,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.137986+0000","last_change":"2024-04-20T02:51:48.253750+0000","last_active":"2024-04-20T02:51:52.137986+0000","last_peered":"2024-04-20T02:51:52.137986+0000","last_clean":"2024-04-20T02:51:52.137986+0000","last_became_active":"2024-04-20T02:50:35.413503+0000","last_became_peered":"2024-04-20T02:50:35.413503+0000","last_unstale":"2024-04-20T02:51:52.137986+0000","last_undegraded":"2024-04-20T02:51:52.137986+0000","last_fullsized":"2024-04-20T02:51:52.137986+0000","mapping_epoch":22,"log_start":"84'50","ondisk_log_start":"84'50","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:57.185462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036879399999999998,"stat_sum":{"num_bytes":5176556,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":124,"num_read_kb":139429,"num_write":32,"num_write_kb":8571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.b","version":"88'148","reported_seq":376,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:54.264918+0000","last_change":"2024-04-20T02:51:48.245992+0000","last_active":"2024-04-20T02:51:54.264918+0000","last_peered":"2024-04-20T02:51:54.264918+0000","last_clean":"2024-04-20T02:51:54.264918+0000","last_became_active":"2024-04-20T02:50:35.242331+0000","last_became_peered":"2024-04-20T02:50:35.242331+0000","last_unstale":"2024-04-20T02:51:54.264918+0000","last_undegraded":"2024-04-20T02:51:54.264918+0000","last_fullsized":"2024-04-20T02:51:54.264918+0000","mapping_epoch":22,"log_start":"86'144","ondisk_log_start":"86'144","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":144,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.773051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065093599999999996,"stat_sum":{"num_bytes":13075813,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":172,"num_read_kb":212900,"num_write":81,"num_write_kb":21776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.c","version":"96'93","reported_seq":279,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-20T02:51:54.305390+0000","last_change":"2024-04-20T02:51:48.241847+0000","last_active":"2024-04-20T02:51:54.305390+0000","last_peered":"2024-04-20T02:51:54.305390+0000","last_clean":"2024-04-20T02:51:54.305390+0000","last_became_active":"2024-04-20T02:50:35.241480+0000","last_became_peered":"2024-04-20T02:50:35.241480+0000","last_unstale":"2024-04-20T02:51:54.305390+0000","last_undegraded":"2024-04-20T02:51:54.305390+0000","last_fullsized":"2024-04-20T02:51:54.305390+0000","mapping_epoch":22,"log_start":"95'89","ondisk_log_start":"95'89","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":89,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:02.915698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031446499999999999,"stat_sum":{"num_bytes":7479115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":49917,"num_write":53,"num_write_kb":14819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.d","version":"96'161","reported_seq":417,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-20T02:51:54.523140+0000","last_change":"2024-04-20T02:51:48.242556+0000","last_active":"2024-04-20T02:51:54.523140+0000","last_peered":"2024-04-20T02:51:54.523140+0000","last_clean":"2024-04-20T02:51:54.523140+0000","last_became_active":"2024-04-20T02:50:35.528107+0000","last_became_peered":"2024-04-20T02:50:35.528107+0000","last_unstale":"2024-04-20T02:51:54.523140+0000","last_undegraded":"2024-04-20T02:51:54.523140+0000","last_fullsized":"2024-04-20T02:51:54.523140+0000","mapping_epoch":22,"log_start":"95'158","ondisk_log_start":"95'158","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":158,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:59.747989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000678117,"stat_sum":{"num_bytes":12099322,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":265,"num_read_kb":295682,"num_write":88,"num_write_kb":23869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.e","version":"92'84","reported_seq":291,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-20T02:51:52.138577+0000","last_change":"2024-04-20T02:51:48.242109+0000","last_active":"2024-04-20T02:51:52.138577+0000","last_peered":"2024-04-20T02:51:52.138577+0000","last_clean":"2024-04-20T02:51:52.138577+0000","last_became_active":"2024-04-20T02:50:35.244872+0000","last_became_peered":"2024-04-20T02:50:35.244872+0000","last_unstale":"2024-04-20T02:51:52.138577+0000","last_undegraded":"2024-04-20T02:51:52.138577+0000","last_fullsized":"2024-04-20T02:51:52.138577+0000","mapping_epoch":22,"log_start":"91'80","ondisk_log_start":"91'80","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'77","last_scrub_stamp":"2024-04-20T02:51:46.088382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:46.088382+0000","objects_scrubbed":3,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:57.901141+0000","scrub_duration":0.002170693,"objects_trimmed":0,"snaptrim_duration":0.00051101800000000002,"stat_sum":{"num_bytes":7008519,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":77784,"num_write":41,"num_write_kb":13592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.f","version":"88'26","reported_seq":212,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.266808+0000","last_change":"2024-04-20T02:51:48.245830+0000","last_active":"2024-04-20T02:51:51.266808+0000","last_peered":"2024-04-20T02:51:51.266808+0000","last_clean":"2024-04-20T02:51:51.266808+0000","last_became_active":"2024-04-20T02:50:35.245323+0000","last_became_peered":"2024-04-20T02:50:35.245323+0000","last_unstale":"2024-04-20T02:51:51.266808+0000","last_undegraded":"2024-04-20T02:51:51.266808+0000","last_fullsized":"2024-04-20T02:51:51.266808+0000","mapping_epoch":22,"log_start":"82'22","ondisk_log_start":"82'22","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":22,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:51:53.742736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014682460000000001,"stat_sum":{"num_bytes":3378207,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":54586,"num_write":19,"num_write_kb":5420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.10","version":"91'103","reported_seq":395,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.150915+0000","last_change":"2024-04-20T02:51:51.150915+0000","last_active":"2024-04-20T02:51:51.150915+0000","last_peered":"2024-04-20T02:51:51.150915+0000","last_clean":"2024-04-20T02:51:51.150915+0000","last_became_active":"2024-04-20T02:51:30.132114+0000","last_became_peered":"2024-04-20T02:51:30.132114+0000","last_unstale":"2024-04-20T02:51:51.150915+0000","last_undegraded":"2024-04-20T02:51:51.150915+0000","last_fullsized":"2024-04-20T02:51:51.150915+0000","mapping_epoch":72,"log_start":"88'98","ondisk_log_start":"88'98","created":60,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'78","last_scrub_stamp":"2024-04-20T02:51:32.278907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.278907+0000","objects_scrubbed":5,"log_size":5,"log_dups_size":96,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:41.876076+0000","scrub_duration":0.0025619089999999998,"objects_trimmed":0,"snaptrim_duration":0.024286707000000001,"stat_sum":{"num_bytes":7243784,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":39422,"num_write":7,"num_write_kb":1804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.11","version":"91'150","reported_seq":508,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-20T02:51:48.273577+0000","last_change":"2024-04-20T02:51:48.242201+0000","last_active":"2024-04-20T02:51:48.273577+0000","last_peered":"2024-04-20T02:51:48.273577+0000","last_clean":"2024-04-20T02:51:48.273577+0000","last_became_active":"2024-04-20T02:51:32.123982+0000","last_became_peered":"2024-04-20T02:51:32.123982+0000","last_unstale":"2024-04-20T02:51:48.273577+0000","last_undegraded":"2024-04-20T02:51:48.273577+0000","last_fullsized":"2024-04-20T02:51:48.273577+0000","mapping_epoch":74,"log_start":"86'147","ondisk_log_start":"86'147","created":60,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'147","last_scrub_stamp":"2024-04-20T02:51:44.085213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:44.085213+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":147,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:49.057479+0000","scrub_duration":0.0022866549999999999,"objects_trimmed":0,"snaptrim_duration":0.00076309100000000003,"stat_sum":{"num_bytes":9109394,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.12","version":"0'0","reported_seq":227,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.266553+0000","last_change":"2024-04-20T02:51:48.244697+0000","last_active":"2024-04-20T02:51:51.266553+0000","last_peered":"2024-04-20T02:51:51.266553+0000","last_clean":"2024-04-20T02:51:51.266553+0000","last_became_active":"2024-04-20T02:51:29.138939+0000","last_became_peered":"2024-04-20T02:51:29.138939+0000","last_unstale":"2024-04-20T02:51:51.266553+0000","last_undegraded":"2024-04-20T02:51:51.266553+0000","last_fullsized":"2024-04-20T02:51:51.266553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:45.458449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:45.458449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:48.851831+0000","scrub_duration":0.0013346709999999999,"objects_trimmed":0,"snaptrim_duration":0.00055967400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":96,"state":"unknown","last_fresh":"2024-04-20T02:51:54.289372+0000","last_change":"2024-04-20T02:51:54.571087+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T02:51:54.289372+0000","last_undegraded":"2024-04-20T02:51:54.289372+0000","last_fullsized":"2024-04-20T02:51:54.289372+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'167","last_scrub_stamp":"2024-04-20T02:51:33.169112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:33.169112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:46.854772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"93'29","reported_seq":213,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.266560+0000","last_change":"2024-04-20T02:51:48.245009+0000","last_active":"2024-04-20T02:51:51.266560+0000","last_peered":"2024-04-20T02:51:51.266560+0000","last_clean":"2024-04-20T02:51:51.266560+0000","last_became_active":"2024-04-20T02:51:30.144817+0000","last_became_peered":"2024-04-20T02:51:30.144817+0000","last_unstale":"2024-04-20T02:51:51.266560+0000","last_undegraded":"2024-04-20T02:51:51.266560+0000","last_fullsized":"2024-04-20T02:51:51.266560+0000","mapping_epoch":72,"log_start":"82'26","ondisk_log_start":"82'26","created":60,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":26,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076120099999999998,"stat_sum":{"num_bytes":1260542,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":39912,"num_write":10,"num_write_kb":2705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.15","version":"93'62","reported_seq":306,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-20T02:51:51.559345+0000","last_change":"2024-04-20T02:51:48.254059+0000","last_active":"2024-04-20T02:51:51.559345+0000","last_peered":"2024-04-20T02:51:51.559345+0000","last_clean":"2024-04-20T02:51:51.559345+0000","last_became_active":"2024-04-20T02:51:32.124965+0000","last_became_peered":"2024-04-20T02:51:32.124965+0000","last_unstale":"2024-04-20T02:51:51.559345+0000","last_undegraded":"2024-04-20T02:51:51.559345+0000","last_fullsized":"2024-04-20T02:51:51.559345+0000","mapping_epoch":74,"log_start":"92'59","ondisk_log_start":"92'59","created":60,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040062300000000002,"stat_sum":{"num_bytes":2066840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":94785,"num_write":28,"num_write_kb":7778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":16438198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"13","length":"1"},{"start":"15","length":"2"},{"start":"18","length":"1"},{"start":"1b","length":"4"},{"start":"27","length":"1"},{"start":"2a","length":"1"}]},{"pgid":"4.16","version":"93'72","reported_seq":323,"reported_epoch":96,"state":"peering","last_fresh":"2024-04-20T02:51:54.290856+0000","last_change":"2024-04-20T02:51:54.291059+0000","last_active":"2024-04-20T02:51:54.290856+0000","last_peered":"2024-04-20T02:51:53.006716+0000","last_clean":"2024-04-20T02:51:27.811685+0000","last_became_active":"2024-04-20T02:51:30.142488+0000","last_became_peered":"2024-04-20T02:51:30.142488+0000","last_unstale":"2024-04-20T02:51:54.290856+0000","last_undegraded":"2024-04-20T02:51:54.290856+0000","last_fullsized":"2024-04-20T02:51:54.290856+0000","mapping_epoch":96,"log_start":"73'67","ondisk_log_start":"73'67","created":60,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":65,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099853799999999999,"stat_sum":{"num_bytes":12201415,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":101,"num_read_kb":121181,"num_write":43,"num_write_kb":12023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":8853601,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":151897090,"num_objects":54,"num_object_clones":16,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":1,"num_read":1351,"num_read_kb":1608649,"num_write":555,"num_write_kb":158465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":36160341,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":33,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335667200,"data_stored":362962947,"data_compressed":1160982,"data_compressed_allocated":995328,"data_compressed_original":2285568,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":104405171,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":520,"num_read_kb":23397,"num_write":704,"num_write_kb":227879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178221056,"data_stored":192647833,"data_compressed":10801746,"data_compressed_allocated":11624448,"data_compressed_original":26533888,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509465,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81572,"kb_used_data":48856,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94290268,"statfs":{"total":96636764160,"available":96553234432,"internally_reserved":0,"allocated":50028544,"data_stored":52103972,"data_compressed":295417,"data_compressed_allocated":294912,"data_compressed_original":1064960,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.379}]},{"osd":1,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.439}]},{"osd":4,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.29699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.318}]}]},{"osd":5,"up_from":38,"seq":163208757261,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80032,"kb_used_data":48212,"kb_used_omap":1,"kb_used_meta":31806,"kb_avail":94291808,"statfs":{"total":96636764160,"available":96554811392,"internally_reserved":0,"allocated":49369088,"data_stored":55463499,"data_compressed":974571,"data_compressed_allocated":1040384,"data_compressed_original":2166784,"omap_allocated":1657,"internal_metadata":32569735},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53168,"kb_used_data":26148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94318672,"statfs":{"total":96636764160,"available":96582320128,"internally_reserved":0,"allocated":26775552,"data_stored":28137854,"data_compressed":893615,"data_compressed_allocated":962560,"data_compressed_original":2162688,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.48999999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.439}]},{"osd":3,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.57599999999999996}]},{"osd":4,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":11.877000000000001,"5min":11.877000000000001,"15min":11.877000000000001},"last":0.623},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":11.805,"5min":11.805,"15min":11.805},"last":0.63900000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.65800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98452,"kb_used_data":67332,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94273388,"statfs":{"total":96636764160,"available":96535949312,"internally_reserved":0,"allocated":68947968,"data_stored":73100220,"data_compressed":926851,"data_compressed_allocated":1003520,"data_compressed_original":2359296,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.316}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.501}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.41999999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.77200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100564,"kb_used_data":69440,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94271276,"statfs":{"total":96636764160,"available":96533786624,"internally_reserved":0,"allocated":71106560,"data_stored":73511414,"data_compressed":2050040,"data_compressed_allocated":2142208,"data_compressed_original":4534272,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.46400000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.496}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.442}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":1.01}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.95299999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.98399999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100120,"kb_used_data":69004,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94271720,"statfs":{"total":96636764160,"available":96534241280,"internally_reserved":0,"allocated":70660096,"data_stored":73804721,"data_compressed":2755679,"data_compressed_allocated":2994176,"data_compressed_original":7258112,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.0369999999999999,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.443}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.496}]},{"osd":3,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":9.9570000000000007,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.52600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.8679999999999999,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.54800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.98,"5min":3.98,"15min":3.98},"last":0.505}]}]},{"osd":0,"up_from":15,"seq":64424509461,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116064,"kb_used_data":83348,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94255776,"statfs":{"total":96636764160,"available":96517914624,"internally_reserved":0,"allocated":85348352,"data_stored":94580661,"data_compressed":2118236,"data_compressed_allocated":2150400,"data_compressed_original":5660672,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.34300000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.39500000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.38400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.437},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.45100000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.46300000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.40899999999999997}]}]},{"osd":4,"up_from":15,"seq":64424509463,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115456,"kb_used_data":84344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94256384,"statfs":{"total":96636764160,"available":96518537216,"internally_reserved":0,"allocated":86368256,"data_stored":90811137,"data_compressed":1840304,"data_compressed_allocated":1986560,"data_compressed_original":4624384,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.255}]},{"osd":1,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.439},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.27200000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.371},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38300000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.43099999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32870400,"data_stored":35403471,"data_compressed":1797788,"data_compressed_allocated":1941504,"data_compressed_original":4558848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13246464,"data_stored":14462667,"data_compressed":842251,"data_compressed_allocated":909312,"data_compressed_original":2162688,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13246464,"data_stored":14462667,"data_compressed":842251,"data_compressed_allocated":909312,"data_compressed_original":2162688,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27971584,"data_stored":29997488,"data_compressed":1759941,"data_compressed_allocated":1880064,"data_compressed_original":3981312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32870400,"data_stored":35403471,"data_compressed":1797788,"data_compressed_allocated":1941504,"data_compressed_original":4558848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14090240,"data_stored":15175321,"data_compressed":974571,"data_compressed_allocated":1040384,"data_compressed_original":2166784,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":43925504,"data_stored":47742748,"data_compressed":2787156,"data_compressed_allocated":3002368,"data_compressed_original":6942720,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53436416,"data_stored":60357894,"data_compressed":357214,"data_compressed_allocated":225280,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54607872,"data_stored":58164017,"data_compressed":84600,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13369344,"data_stored":14190393,"data_compressed":54370,"data_compressed_allocated":57344,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45625344,"data_stored":47181133,"data_compressed":290099,"data_compressed_allocated":262144,"data_compressed_original":552960,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57208832,"data_stored":63060483,"data_compressed":84600,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33013760,"data_stored":37670692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":29523968,"data_stored":31236827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":48881664,"data_stored":51101508,"data_compressed":290099,"data_compressed_allocated":262144,"data_compressed_original":552960,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:51:57.543 INFO:tasks.ceph.ceph_manager.ceph:decrease pool size by 10 2024-04-20T02:51:57.543 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 16 2024-04-20T02:51:57.578 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:57.578 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:51:57.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:51:57.578 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:51:57.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.634312011000006, 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:57.584 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.626833692999995, 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:51:57.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:51:57.589 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.590 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.590 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.590 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:51:57.590 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:57.590 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.622075793999997, 2024-04-20T02:51:57.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:51:57.590 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 63.198646859999997, 2024-04-20T02:51:57.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:57.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:51:57.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:51:57.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.606 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 63.198609976999997, 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:57.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:51:57.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 63.198585344000001, 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:57.613 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:51:57.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:51:57.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.617 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 63.192091933, 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.618 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 63.186027910999996, 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.624 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:51:57.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:51:57.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:51:57.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 63.175966146, 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.635 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 63.175273541999999, 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:51:57.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.641 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 63.175261888000001, 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:51:57.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.469197332, 2024-04-20T02:51:57.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.450136616000002, 2024-04-20T02:51:57.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:51:57.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:51:57.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:51:57.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.444190302000003, 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:51:57.656 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.443624861000004, 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:51:57.662 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:57.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:51:57.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:51:57.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:51:57.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:51:57.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.443598829999999, 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.668 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:51:57.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:51:57.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:51:57.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:51:57.672 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.441386029, 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:51:57.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:51:57.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:51:57.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.4360365050000001, 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:51:57.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:51:57.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:51:57.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:51:57.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.4355665479999999, 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:51:57.681 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.685 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.435544012, 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:51:57.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:51:57.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:51:57.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:51:57.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:51:57.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.690 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:51:57.690 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:51:57.690 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:57.691 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:51:57.692 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:57.692+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:57.693 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:51:57.792 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:57.793+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:57.824 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:51:57.824 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:51:57.824 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:51:57.824 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:51:57.835 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:51:57.845 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:51:57.864 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:51:57.883 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:51:57.902 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:51:57.919 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:57.920+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:57.921 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:51:57.940 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:51:57.961 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:51:57.983 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:51:57.983 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:51:57.983 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:51:57.983 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:51:57.983 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:51:57.984 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:51:57.984 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:51:57.984 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:51:57.984 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:51:57.994 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:57.994+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:58.007 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:58.007 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:58.017 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:51:58.095 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:58.096+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:58.133 INFO:teuthology.orchestra.run.smithi105.stderr:set pool 4 pg_num to 16 2024-04-20T02:51:58.134 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:51:58.196 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:51:58.196+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:51:58.216 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:58.216 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:58.227 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:51:58.296 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:58.297+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:58.363 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:58.363 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:58.374 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:51:58.415 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:58.416+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:58.498 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:58.499+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:58.541 INFO:tasks.rados.rados.0.smithi105.stdout:1336: set_chunk oid 15 offset: 1375744 length: 1486336 target oid 1ec8dd92609e9e9f3803d04de0bd5fb3b223ada14e5a58ec1de59e1fd4f06a73 offset: 0 snap -1 2024-04-20T02:51:58.541 INFO:tasks.rados.rados.0.smithi105.stdout:1337: write oid 40 current snap is 22 2024-04-20T02:51:58.541 INFO:tasks.rados.rados.0.smithi105.stdout:1330: finishing write tid 3 to smithi10548823-36 2024-04-20T02:51:58.541 INFO:tasks.rados.rados.0.smithi105.stdout:1330: finishing write tid 4 to smithi10548823-36 2024-04-20T02:51:58.541 INFO:tasks.rados.rados.0.smithi105.stdout:1330: oid 36 updating version 0 to 277 2024-04-20T02:51:58.541 INFO:tasks.rados.rados.0.smithi105.stdout:1330: oid 36 updating version 277 to 278 2024-04-20T02:51:58.541 INFO:tasks.rados.rados.0.smithi105.stdout:1330: oid 36 updating version 278 to 279 2024-04-20T02:51:58.541 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 279 (ObjNum 244 snap 22 seq_num 244) dirty exists 2024-04-20T02:51:58.541 INFO:tasks.rados.rados.0.smithi105.stdout:1330: left oid 36 (ObjNum 244 snap 22 seq_num 244) 2024-04-20T02:51:58.541 INFO:tasks.rados.rados.0.smithi105.stdout:1337: seq_num 248 ranges {568136=785048,1929299=238837} 2024-04-20T02:51:58.547 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:58.547 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:58.548 INFO:tasks.rados.rados.0.smithi105.stdout:1337: writing smithi10548823-40 from 568136 to 1353184 tid 1 2024-04-20T02:51:58.552 INFO:tasks.rados.rados.0.smithi105.stdout:1337: writing smithi10548823-40 from 1929299 to 2168136 tid 2 2024-04-20T02:51:58.552 INFO:tasks.rados.rados.0.smithi105.stdout:1330: done (8 left) 2024-04-20T02:51:58.553 INFO:tasks.rados.rados.0.smithi105.stdout:1338: write oid 1 current snap is 22 2024-04-20T02:51:58.553 INFO:tasks.rados.rados.0.smithi105.stdout:1338: seq_num 249 ranges {584943=460297,1587781=597162} 2024-04-20T02:51:58.558 INFO:tasks.rados.rados.0.smithi105.stdout:1338: writing smithi10548823-1 from 584943 to 1045240 tid 1 2024-04-20T02:51:58.559 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:51:58.566 INFO:tasks.rados.rados.0.smithi105.stdout:1338: writing smithi10548823-1 from 1587781 to 2184943 tid 2 2024-04-20T02:51:58.566 INFO:tasks.rados.rados.0.smithi105.stdout:1339: write oid 3 current snap is 22 2024-04-20T02:51:58.566 INFO:tasks.rados.rados.0.smithi105.stdout:1339: seq_num 250 ranges {601750=535546,1929910=271840} 2024-04-20T02:51:58.573 INFO:tasks.rados.rados.0.smithi105.stdout:1339: writing smithi10548823-3 from 601750 to 1137296 tid 1 2024-04-20T02:51:58.580 INFO:tasks.rados.rados.0.smithi105.stdout:1339: writing smithi10548823-3 from 1929910 to 2201750 tid 2 2024-04-20T02:51:58.580 INFO:tasks.rados.rados.0.smithi105.stdout:1340: read oid 47 snap -1 2024-04-20T02:51:58.580 INFO:tasks.rados.rados.0.smithi105.stdout:1340: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:51:58.580 INFO:tasks.rados.rados.0.smithi105.stdout:1341: read oid 43 snap -1 2024-04-20T02:51:58.580 INFO:tasks.rados.rados.0.smithi105.stdout:1341: expect (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:51:58.581 INFO:tasks.rados.rados.0.smithi105.stdout:1332: expect (ObjNum 210 snap 19 seq_num 210) 2024-04-20T02:51:58.599 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:51:58.600+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:51:58.693 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:58.693 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:58.700 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:58.701+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:58.703 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:51:58.758 INFO:tasks.rados.rados.0.smithi105.stdout:1333: finishing write tid 1 to smithi10548823-22 2024-04-20T02:51:58.758 INFO:tasks.rados.rados.0.smithi105.stdout:1333: finishing write tid 2 to smithi10548823-22 2024-04-20T02:51:58.758 INFO:tasks.rados.rados.0.smithi105.stdout:1333: finishing write tid 3 to smithi10548823-22 2024-04-20T02:51:58.758 INFO:tasks.rados.rados.0.smithi105.stdout:1333: finishing write tid 4 to smithi10548823-22 2024-04-20T02:51:58.758 INFO:tasks.rados.rados.0.smithi105.stdout:1333: oid 22 updating version 0 to 83 2024-04-20T02:51:58.758 INFO:tasks.rados.rados.0.smithi105.stdout:1333: oid 22 version 83 is already newer than 82 2024-04-20T02:51:58.758 INFO:tasks.rados.rados.0.smithi105.stdout:1333: oid 22 version 83 is already newer than 81 2024-04-20T02:51:58.758 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 83 (ObjNum 246 snap 22 seq_num 246) dirty exists 2024-04-20T02:51:58.758 INFO:tasks.rados.rados.0.smithi105.stdout:1333: left oid 22 (ObjNum 246 snap 22 seq_num 246) 2024-04-20T02:51:58.758 INFO:tasks.rados.rados.0.smithi105.stdout:1335: expect (ObjNum 222 snap 21 seq_num 222) 2024-04-20T02:51:58.801 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:58.801+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:58.872 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:58.873 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:58.883 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:51:58.902 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:51:58.902+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:51:59.003 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:59.003+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:59.010 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:59.010 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:59.015 INFO:tasks.rados.rados.0.smithi105.stdout:1334: finishing write tid 1 to smithi10548823-27 2024-04-20T02:51:59.015 INFO:tasks.rados.rados.0.smithi105.stdout:1334: finishing write tid 2 to smithi10548823-27 2024-04-20T02:51:59.015 INFO:tasks.rados.rados.0.smithi105.stdout:1342: read oid 39 snap -1 2024-04-20T02:51:59.015 INFO:tasks.rados.rados.0.smithi105.stdout:1342: expect (ObjNum 235 snap 22 seq_num 235) 2024-04-20T02:51:59.015 INFO:tasks.rados.rados.0.smithi105.stdout:1332: done (12 left) 2024-04-20T02:51:59.015 INFO:tasks.rados.rados.0.smithi105.stdout:1333: done (11 left) 2024-04-20T02:51:59.015 INFO:tasks.rados.rados.0.smithi105.stdout:1335: done (10 left) 2024-04-20T02:51:59.015 INFO:tasks.rados.rados.0.smithi105.stdout:1343: write (excl) oid 19 current snap is 22 2024-04-20T02:51:59.015 INFO:tasks.rados.rados.0.smithi105.stdout:1334: finishing write tid 3 to smithi10548823-27 2024-04-20T02:51:59.015 INFO:tasks.rados.rados.0.smithi105.stdout:1334: finishing write tid 4 to smithi10548823-27 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1334: oid 27 updating version 0 to 151 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1334: oid 27 version 151 is already newer than 150 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1334: oid 27 updating version 151 to 152 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 152 (ObjNum 247 snap 22 seq_num 247) dirty exists 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1334: left oid 27 (ObjNum 247 snap 22 seq_num 247) 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1329: finishing write tid 1 to smithi10548823-8 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1329: finishing write tid 2 to smithi10548823-8 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1329: finishing write tid 3 to smithi10548823-8 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1329: finishing write tid 4 to smithi10548823-8 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1329: oid 8 updating version 0 to 77 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1329: oid 8 version 77 is already newer than 76 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1329: oid 8 updating version 77 to 78 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 78 (ObjNum 243 snap 22 seq_num 243) dirty exists 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1329: left oid 8 (ObjNum 243 snap 22 seq_num 243) 2024-04-20T02:51:59.016 INFO:tasks.rados.rados.0.smithi105.stdout:1331: finishing write tid 1 to smithi10548823-17 2024-04-20T02:51:59.017 INFO:tasks.rados.rados.0.smithi105.stdout:1343: seq_num 251 ranges {618557=727148,2104745=113812} 2024-04-20T02:51:59.020 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:51:59.022 INFO:tasks.rados.rados.0.smithi105.stdout:1343: writing smithi10548823-19 from 618557 to 1345705 tid 1 2024-04-20T02:51:59.026 INFO:tasks.rados.rados.0.smithi105.stdout:1343: writing smithi10548823-19 from 2104745 to 2218557 tid 2 2024-04-20T02:51:59.026 INFO:tasks.rados.rados.0.smithi105.stdout:1329: done (10 left) 2024-04-20T02:51:59.026 INFO:tasks.rados.rados.0.smithi105.stdout:1334: done (9 left) 2024-04-20T02:51:59.027 INFO:tasks.rados.rados.0.smithi105.stdout:1344: snap_remove snap 21 2024-04-20T02:51:59.104 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:51:59.104+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:51:59.152 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:59.152 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:59.162 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:51:59.205 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:59.206+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:59.284 INFO:tasks.rados.rados.0.smithi105.stdout:1344: done (9 left) 2024-04-20T02:51:59.284 INFO:tasks.rados.rados.0.smithi105.stdout:1345: set_chunk oid 20 target oid 2024-04-20T02:51:59.303 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:59.304 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:59.306 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:59.306+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:59.315 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:51:59.348 INFO:tasks.rados.rados.0.smithi105.stdout:1345: set_chunk oid 20 offset: 13824 length: 2550272 target oid f9ba5b4a6de3a736e4405528cc5891f590f1fb61b1600aeb2938b319332d1605 offset: 0 snap -1 2024-04-20T02:51:59.348 INFO:tasks.rados.rados.0.smithi105.stdout:1346: set_chunk oid 6 target oid 2024-04-20T02:51:59.350 INFO:tasks.rados.rados.0.smithi105.stdout:1346: set_chunk oid 6 offset: 57856 length: 48640 target oid 0aba7d2ef8f6230dba5b113b61d0d5934ef45dfddb0085cb33f64bfd71531bf9 offset: 0 snap -1 2024-04-20T02:51:59.350 INFO:tasks.rados.rados.0.smithi105.stdout:1347: set_chunk oid 24 target oid 2024-04-20T02:51:59.407 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:51:59.408+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:51:59.446 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:59.447 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:59.457 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:51:59.508 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:59.508+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:59.608 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:59.609+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:59.625 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:59.625 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:59.636 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:51:59.709 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:51:59.710+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:51:59.764 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:59.764 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:59.774 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:51:59.810 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:51:59.811+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:51:59.912 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:51:59.912+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:51:59.941 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:51:59.942 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:51:59.952 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:51:59.993 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:00.039 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:00.040+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:00.086 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:00.087 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:00.097 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:00.139 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:00.140+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:00.144 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:00.151 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:52:00.168 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:00.189 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:00.209 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:00.215 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:00.215+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:00.231 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:00.250 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:00.271 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:00.274 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:00.274 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:00.274 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:00.274 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:00.286 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:00.304 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:00.309 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:00.309 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:00.309 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:00.309 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:00.309 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:00.309 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:00.309 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:00.309 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:00.316 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:00.316+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:00.417 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:00.417+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:00.436 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:00.436 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:00.436 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:00.436 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:00.436 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:00.446 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:00.518 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:00.519+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:00.576 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:00.576 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.630825891000001, 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041399200999999997, 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945227+0000", 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947429+0000", 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022028989999999999 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947448+0000", 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.821e-05 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947460+0000", 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2373e-05 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947511+0000", 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0844999999999999e-05 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947531+0000", 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0017000000000002e-05 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986580+0000", 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039049213999999999 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986627+0000", 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7070000000000002e-05 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1145 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'30, mlcod=34'26)", 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.612095191999998, 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026752594000000001, 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963958+0000", 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964662+0000", 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070467299999999995 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964678+0000", 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5597999999999999e-05 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964687+0000", 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2979999999999997e-06 2024-04-20T02:52:00.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968184+0000", 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034972250000000001 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968211+0000", 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6829999999999999e-05 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990640+0000", 2024-04-20T02:52:00.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022429317000000001 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990711+0000", 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0794e-05 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:129 3.0 e42/21 3:0256f537:::3f82a762bf2204de901e10565f70e759ee2a26bf55ffef4c8d5027a47e96ec31:head v 42'78, mlcod=42'76)", 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:52:00.585 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.441406616999998, 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024418002000000001, 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134647+0000", 2024-04-20T02:52:00.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134646+0000", 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012265099999999999 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134789+0000", 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9924000000000001e-05 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134799+0000", 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0042e-05 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134842+0000", 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3791000000000001e-05 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134866+0000", 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3791e-05 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159017+0000", 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024150919 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.159065+0000", 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8201000000000001e-05 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.15:134 3.0 e42/21 3:33aaac3f:::d0aca4dab23a9f22fb58211e350c7c1c1b7601bafbb496b93afbbc91e2ed921c:head v 42'81, mlcod=42'76)", 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.431384418, 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024591152000000002, 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144669+0000", 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144949+0000", 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028058500000000002 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144959+0000", 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0335e-05 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.144967+0000", 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6040000000000004e-06 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151637+0000", 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066694670000000001 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.151659+0000", 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2855000000000001e-05 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169224+0000", 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017565041 2024-04-20T02:52:00.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.169261+0000", 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6179e-05 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1462 3.0 e44/21 3:29b349f1:::ea1112aea5e7be379214f0429a6e898cf4f4a05bdbb29b6cd15764b4776418b1:head v 44'85, mlcod=42'83)", 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.398264533000003, 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034060275000000001, 2024-04-20T02:52:00.594 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178932+0000", 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011426069999999999 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178944+0000", 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.287e-05 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178962+0000", 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7669999999999999e-05 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179013+0000", 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1295000000000003e-05 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179038+0000", 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4859999999999999e-05 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211813+0000", 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032775190000000003 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211850+0000", 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6216e-05 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.598 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1629 3.0 e51/21 3:088cc2f3:::a1744cc895df5e9d3e624b281fcb9edc57f3f342d1f108d4c38dc778f219a41a:head v 51'93, mlcod=45'91)", 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.772965196000001, 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035323188999999998, 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:52:00.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803088+0000", 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804187+0000", 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00109891 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804201+0000", 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3772e-05 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804213+0000", 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2335e-05 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804285+0000", 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1271000000000003e-05 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804306+0000", 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1359999999999999e-05 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838377+0000", 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034070994 2024-04-20T02:52:00.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838412+0000", 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4790999999999999e-05 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1653 4.6 e51/22 4:697da67d:::smithi10548823-48:head v 51'41, mlcod=45'37)", 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.151226088000001, 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037653640000000002, 2024-04-20T02:52:00.603 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424827+0000", 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425066+0000", 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023957399999999999 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425117+0000", 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1082999999999999e-05 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425146+0000", 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8129000000000001e-05 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425217+0000", 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1402000000000005e-05 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425278+0000", 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1498e-05 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462456+0000", 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037177106000000001 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462481+0000", 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5851e-05 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1786 4.d e56/22 4:b376c20e:::smithi10548823-49:head v 56'80, mlcod=55'77)", 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:52:00.607 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.764134550999998, 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028136330000000001, 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812644+0000", 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072510100000000002 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812656+0000", 2024-04-20T02:52:00.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1688999999999999e-05 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812663+0000", 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812707+0000", 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4573999999999997e-05 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812722+0000", 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5319e-05 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840021+0000", 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027298111999999999 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840056+0000", 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5086000000000003e-05 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.611 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1799 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'53, mlcod=54'49)", 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.501805529999999, 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050527214000000001, 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074614+0000", 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036631499999999998 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074624+0000", 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8330000000000006e-06 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074632+0000", 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3599999999999998e-06 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074683+0000", 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1137999999999997e-05 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074704+0000", 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1616000000000001e-05 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124714+0000", 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050009642 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124776+0000", 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1581000000000006e-05 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.615 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.487309433999997, 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046184680999999998, 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:52:00.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:52:00.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088743+0000", 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090327+0000", 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001584071 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090343+0000", 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5193e-05 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090354+0000", 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1336e-05 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090411+0000", 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6589999999999999e-05 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090431+0000", 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0579000000000001e-05 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134883+0000", 2024-04-20T02:52:00.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044451877000000001 2024-04-20T02:52:00.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134929+0000", 2024-04-20T02:52:00.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6131000000000003e-05 2024-04-20T02:52:00.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.620 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1800 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'54, mlcod=54'49)", 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.474903359999999, 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027029681999999999, 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101150+0000", 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101284+0000", 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000134688 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101295+0000", 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101305+0000", 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0930999999999999e-05 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101347+0000", 2024-04-20T02:52:00.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1522e-05 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101369+0000", 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2027000000000001e-05 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128138+0000", 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026768626 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128180+0000", 2024-04-20T02:52:00.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2568999999999998e-05 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1807 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'87, mlcod=56'82)", 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.467222534000001, 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026347280000000001, 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108831+0000", 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.108830+0000", 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109221+0000", 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039087699999999998 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109239+0000", 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8904000000000001e-05 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.109271+0000", 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1319000000000003e-05 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120423+0000", 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011151912999999999 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.120454+0000", 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1464999999999998e-05 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135158+0000", 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014703554000000001 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.135179+0000", 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0965999999999999e-05 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2038 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'159, mlcod=0'0)", 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 35.994433960999999, 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044306238999999997, 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:52:00.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581964+0000", 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034420099999999998 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581980+0000", 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6578999999999998e-05 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581991+0000", 2024-04-20T02:52:00.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0949e-05 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582734+0000", 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074277299999999996 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582761+0000", 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6627000000000001e-05 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625882+0000", 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043121617000000001 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625926+0000", 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3838999999999998e-05 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2039 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'160, mlcod=0'0)", 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:52:00.633 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 35.974035876000002, 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027318719000000002, 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602017+0000", 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602364+0000", 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000347139 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602375+0000", 2024-04-20T02:52:00.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.128e-05 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602384+0000", 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2600000000000005e-06 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605223+0000", 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0028393400000000001 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605245+0000", 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2078000000000001e-05 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629297+0000", 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024051605 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629337+0000", 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0126000000000002e-05 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.637 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2137 3.0 e70/21 3:0d26903d:::7146376437549e0b99264d159bf1e9c8bc43ac1c7eb519f65899cfd592a43a4a:head v 70'98, mlcod=55'96)", 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.791907891999998, 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024394852000000002, 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784146+0000", 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.784144+0000", 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785280+0000", 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001135741 2024-04-20T02:52:00.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785299+0000", 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8845999999999999e-05 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785310+0000", 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1450999999999999e-05 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785362+0000", 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1798000000000001e-05 2024-04-20T02:52:00.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.785388+0000", 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5763999999999999e-05 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808507+0000", 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023118742000000001 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.808541+0000", 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4258000000000003e-05 2024-04-20T02:52:00.641 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2142 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'65, mlcod=66'61)", 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.759208479000002, 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033365672999999998, 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:52:00.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816981+0000", 2024-04-20T02:52:00.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135952 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816994+0000", 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3405e-05 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817003+0000", 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4319999999999999e-06 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817055+0000", 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1842999999999998e-05 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817076+0000", 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1375000000000001e-05 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850179+0000", 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033103051000000001 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850211+0000", 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1896000000000001e-05 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.434673864000001, 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039421115999999999, 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141379+0000", 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142797+0000", 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001418702 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142816+0000", 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8907999999999999e-05 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142825+0000", 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1020000000000003e-06 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142872+0000", 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6168999999999998e-05 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142891+0000", 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9561000000000001e-05 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180704+0000", 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037812643 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180801+0000", 2024-04-20T02:52:00.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7361999999999994e-05 2024-04-20T02:52:00.650 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.650 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.650 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.650 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2280 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'105, mlcod=73'100)", 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.413992586999999, 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029657706999999998, 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:52:00.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162060+0000", 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162455+0000", 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039569799999999999 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162472+0000", 2024-04-20T02:52:00.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6820000000000002e-05 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162485+0000", 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2388999999999999e-05 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163758+0000", 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001273906 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163784+0000", 2024-04-20T02:52:00.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5352000000000001e-05 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191663+0000", 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027879583999999999 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191719+0000", 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640000000000003e-05 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2559 3.0 e91/21 3:0870df8d:::348e4a2a55868b09db2e6d173951b4676cfef7376c5a81f7a82bb8f7ccc4ada5:head v 91'132, mlcod=91'130)", 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.107236825999999, 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032176705999999999, 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468815+0000", 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469831+0000", 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001015076 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469853+0000", 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2713000000000001e-05 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469865+0000", 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1904e-05 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469910+0000", 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5114999999999997e-05 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469934+0000", 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4133000000000001e-05 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.500951+0000", 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031016851000000002 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.500994+0000", 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2496000000000001e-05 2024-04-20T02:52:00.658 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2788 4.14 e98/97 4:2e88ea27:::smithi10548823-12:head v 98'31, mlcod=0'0)", 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:56.144245+0000", 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 4.4318091620000004, 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038004836, 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:00.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144245+0000", 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144245+0000", 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144244+0000", 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:00.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144489+0000", 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024499599999999998 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144512+0000", 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2989e-05 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144521+0000", 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2769999999999994e-06 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144580+0000", 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9119000000000001e-05 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144612+0000", 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1962000000000003e-05 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.182209+0000", 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037597034000000001 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:00.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:00.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:00.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.182250+0000", 2024-04-20T02:52:00.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0376000000000001e-05 2024-04-20T02:52:00.663 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.663 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.663 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.663 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:00.663 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:00.663 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:00.665 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:00.665 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:00.665+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:00.766 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:00.766+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:00.833 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:00.833 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:00.833 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:00.833 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:00.843 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:00.866 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:00.867+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:00.967 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:00.968+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:00.970 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:00.970 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:00.970 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:00.981 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:01.068 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:01.070+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:01.132 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:52:01.166 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:01.166 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:01.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:01.166 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 74.620757901000005, 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:52:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:52:01.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 74.614536826999995, 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:52:01.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:52:01.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:52:01.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:52:01.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:52:01.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 74.606136622999998, 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.179 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:52:01.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 74.605793977999994, 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.185 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:52:01.189 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 66.799471975000003, 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:52:01.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:52:01.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 66.792209013000004, 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:01.201 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 66.786266483999995, 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:52:01.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 66.786063491999997, 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.207 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:01.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:52:01.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 66.786040564999993, 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.213 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 66.775956831000002, 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:52:01.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:52:01.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:52:01.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 66.759071422999995, 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 66.742794982000007, 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:52:01.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.718239509999997, 2024-04-20T02:52:01.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:52:01.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 36.403923734000003, 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:52:01.236 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2545 4.9 e91/81 4:95e19a88:::smithi10548823-40:head v 91'253, mlcod=91'251)", 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.749286618999999, 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054757404000000003, 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.242 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415210+0000", 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416004+0000", 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079381199999999997 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416022+0000", 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7969e-05 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416031+0000", 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9279999999999999e-06 2024-04-20T02:52:01.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416100+0000", 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8891000000000002e-05 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416128+0000", 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8203000000000001e-05 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469912+0000", 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053783794000000003 2024-04-20T02:52:01.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469968+0000", 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6474000000000002e-05 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.8162578119999999, 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:52:01.246 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:52:01.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:52:01.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:52:01.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:52:01.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2700 4.0 4:007b2a87:::smithi10548823-39:head [write 1967200~714747] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.8077676179999997, 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061846287, 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2700 2024-04-20T02:52:01.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356728+0000", 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:01.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356879+0000", 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000150958 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356894+0000", 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5132e-05 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356906+0000", 2024-04-20T02:52:01.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2071e-05 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366666+0000", 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0097595289999999994 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366774+0000", 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108527 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366956+0000", 2024-04-20T02:52:01.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181359 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390515+0000", 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023559902000000001 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406790+0000", 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016274555999999999 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418488+0000", 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011697591 2024-04-20T02:52:01.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418529+0000", 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1545000000000003e-05 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418576+0000", 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6903000000000001e-05 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2701 4.0 4:007b2a87:::smithi10548823-39:head [write 3421983~426820] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.8005916259999992, 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054799176999999998, 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2701 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363904+0000", 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364154+0000", 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000250179 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364171+0000", 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6748e-05 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364182+0000", 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1371e-05 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390552+0000", 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026369607 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390652+0000", 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010059499999999999 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390857+0000", 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020445199999999999 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418224+0000", 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027367316999999999 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418282+0000", 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8279000000000002e-05 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418618+0000", 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000335967 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418661+0000", 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2614999999999998e-05 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418705+0000", 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3993999999999999e-05 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.263 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2702 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (57) in=7b,truncate 3848803] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.8003055719999992, 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058669165000000002, 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2702 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364191+0000", 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364196+0000", 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4490000000000001e-06 2024-04-20T02:52:01.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364201+0000", 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5040000000000002e-06 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364206+0000", 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.405895+0000", 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041689047 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406021+0000", 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000125561 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406541+0000", 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052000699999999996 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418451+0000", 2024-04-20T02:52:01.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011910041999999999 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418747+0000", 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029622399999999999 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422704+0000", 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039571859999999997 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422781+0000", 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.661e-05 2024-04-20T02:52:01.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422861+0000", 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9987999999999994e-05 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2703 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.8002865200000002, 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058940278999999998, 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:01.269 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2703 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:01.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364213+0000", 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4830000000000002e-06 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364220+0000", 2024-04-20T02:52:01.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.007e-06 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364225+0000", 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0350000000000001e-06 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406820+0000", 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042595135999999999 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406923+0000", 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102846 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422897+0000", 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015974202999999999 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423015+0000", 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117581 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:01.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423151+0000", 2024-04-20T02:52:01.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013635899999999999 2024-04-20T02:52:01.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.274 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.274 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:01.274 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:01.274 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:01.276 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:01.277 INFO:tasks.rados.rados.0.smithi105.stdout:1347: set_chunk oid 24 offset: 2390016 length: 1359360 target oid 65c4a6cbe46e8d949445597300a5482d595a2e84e6ac0aa957f292131d610636 offset: 0 snap -1 2024-04-20T02:52:01.277 INFO:tasks.rados.rados.0.smithi105.stdout:1348: set_chunk oid 25 target oid 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1331: finishing write tid 2 to smithi10548823-17 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1331: finishing write tid 3 to smithi10548823-17 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1331: finishing write tid 4 to smithi10548823-17 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1331: oid 17 updating version 0 to 80 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1331: oid 17 updating version 80 to 81 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1331: oid 17 version 81 is already newer than 79 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 81 (ObjNum 245 snap 22 seq_num 245) dirty exists 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1331: left oid 17 (ObjNum 245 snap 22 seq_num 245) 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1336: finishing set_chunk to oid 15 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 268 (ObjNum 231 snap 22 seq_num 231) dirty exists 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1348: set_chunk oid 25 offset: 1980928 length: 550400 target oid 8a8e69d0663bc7a8f02255918ddb1236ba7e1d1be618d63a57ef6a89adcecb1b offset: 0 snap -1 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1331: done (12 left) 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1336: done (11 left) 2024-04-20T02:52:01.278 INFO:tasks.rados.rados.0.smithi105.stdout:1349: read oid 34 snap -1 2024-04-20T02:52:01.279 INFO:tasks.rados.rados.0.smithi105.stdout:1349: expect (ObjNum 214 snap 20 seq_num 214) 2024-04-20T02:52:01.279 INFO:tasks.rados.rados.0.smithi105.stdout:1350: read oid 18 snap -1 2024-04-20T02:52:01.279 INFO:tasks.rados.rados.0.smithi105.stdout:1350: expect (ObjNum 237 snap 22 seq_num 237) 2024-04-20T02:52:01.279 INFO:tasks.rados.rados.0.smithi105.stdout:1351: set_chunk oid 8 target oid 2024-04-20T02:52:01.279 INFO:tasks.rados.rados.0.smithi105.stdout:1351: set_chunk oid 8 offset: 374272 length: 77824 target oid 114a7d83274be10327604ee62ec78adf151dd24db77969982a0472911d70b068 offset: 0 snap -1 2024-04-20T02:52:01.279 INFO:tasks.rados.rados.0.smithi105.stdout:1352: set_chunk oid 29 target oid 2024-04-20T02:52:01.279 INFO:tasks.rados.rados.0.smithi105.stdout:1352: set_chunk oid 29 offset: 610304 length: 837632 target oid c1fa0a4063dd932bfd5e32f333ea310d472a0af833f2c4fa97b46331bcc73e3d offset: 0 snap -1 2024-04-20T02:52:01.279 INFO:tasks.rados.rados.0.smithi105.stdout:1353: write (excl) oid 17 current snap is 22 2024-04-20T02:52:01.279 INFO:tasks.rados.rados.0.smithi105.stdout:1338: finishing write tid 1 to smithi10548823-1 2024-04-20T02:52:01.279 INFO:tasks.rados.rados.0.smithi105.stdout:1341: expect (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:52:01.279 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:01.276+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:01.376 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:01.378+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:01.409 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:01.410 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:01.410 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:01.410 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:01.420 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:01.477 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:01.478+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:01.523 INFO:tasks.rados.rados.0.smithi105.stdout:1340: expect (ObjNum 46 snap 0 seq_num 46) 2024-04-20T02:52:01.549 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:01.549 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:01.549 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:01.549 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:01.549 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:01.559 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:01.600 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:01.601+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:01.614 INFO:tasks.rados.rados.0.smithi105.stdout:1337: finishing write tid 1 to smithi10548823-40 2024-04-20T02:52:01.614 INFO:tasks.rados.rados.0.smithi105.stdout:1337: finishing write tid 2 to smithi10548823-40 2024-04-20T02:52:01.614 INFO:tasks.rados.rados.0.smithi105.stdout:1337: finishing write tid 3 to smithi10548823-40 2024-04-20T02:52:01.614 INFO:tasks.rados.rados.0.smithi105.stdout:1337: finishing write tid 4 to smithi10548823-40 2024-04-20T02:52:01.614 INFO:tasks.rados.rados.0.smithi105.stdout:1337: oid 40 updating version 0 to 283 2024-04-20T02:52:01.614 INFO:tasks.rados.rados.0.smithi105.stdout:1337: oid 40 updating version 283 to 284 2024-04-20T02:52:01.614 INFO:tasks.rados.rados.0.smithi105.stdout:1337: oid 40 version 284 is already newer than 282 2024-04-20T02:52:01.614 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 284 (ObjNum 248 snap 22 seq_num 248) dirty exists 2024-04-20T02:52:01.614 INFO:tasks.rados.rados.0.smithi105.stdout:1337: left oid 40 (ObjNum 248 snap 22 seq_num 248) 2024-04-20T02:52:01.615 INFO:tasks.rados.rados.0.smithi105.stdout:1338: finishing write tid 2 to smithi10548823-1 2024-04-20T02:52:01.615 INFO:tasks.rados.rados.0.smithi105.stdout:1353: seq_num 252 ranges {635364=402397,1763227=472137} 2024-04-20T02:52:01.619 INFO:tasks.rados.rados.0.smithi105.stdout:1353: writing smithi10548823-17 from 635364 to 1037761 tid 1 2024-04-20T02:52:01.626 INFO:tasks.rados.rados.0.smithi105.stdout:1353: writing smithi10548823-17 from 1763227 to 2235364 tid 2 2024-04-20T02:52:01.626 INFO:tasks.rados.rados.0.smithi105.stdout:1337: done (15 left) 2024-04-20T02:52:01.626 INFO:tasks.rados.rados.0.smithi105.stdout:1340: done (14 left) 2024-04-20T02:52:01.626 INFO:tasks.rados.rados.0.smithi105.stdout:1341: done (13 left) 2024-04-20T02:52:01.626 INFO:tasks.rados.rados.0.smithi105.stdout:1354: read oid 48 snap -1 2024-04-20T02:52:01.626 INFO:tasks.rados.rados.0.smithi105.stdout:1354: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:01.626 INFO:tasks.rados.rados.0.smithi105.stdout:1355: tier_promote oid 13 2024-04-20T02:52:01.627 INFO:tasks.rados.rados.0.smithi105.stdout:1338: finishing write tid 3 to smithi10548823-1 2024-04-20T02:52:01.627 INFO:tasks.rados.rados.0.smithi105.stdout:1338: finishing write tid 4 to smithi10548823-1 2024-04-20T02:52:01.627 INFO:tasks.rados.rados.0.smithi105.stdout:1338: oid 1 updating version 0 to 235 2024-04-20T02:52:01.627 INFO:tasks.rados.rados.0.smithi105.stdout:1338: oid 1 version 235 is already newer than 234 2024-04-20T02:52:01.627 INFO:tasks.rados.rados.0.smithi105.stdout:1338: oid 1 version 235 is already newer than 233 2024-04-20T02:52:01.627 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 235 (ObjNum 249 snap 22 seq_num 249) dirty exists 2024-04-20T02:52:01.627 INFO:tasks.rados.rados.0.smithi105.stdout:1338: left oid 1 (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:01.627 INFO:tasks.rados.rados.0.smithi105.stdout:1339: finishing write tid 1 to smithi10548823-3 2024-04-20T02:52:01.627 INFO:tasks.rados.rados.0.smithi105.stdout:1338: done (14 left) 2024-04-20T02:52:01.627 INFO:tasks.rados.rados.0.smithi105.stdout:1356: set_chunk oid 27 target oid 2024-04-20T02:52:01.655 INFO:tasks.rados.rados.0.smithi105.stdout:1356: set_chunk oid 27 offset: 709120 length: 778240 target oid 1b5b5e25ee11cf0587074eaaf93a5a4a892d34f1d55489c1e81c6af87d3c69e6 offset: 0 snap -1 2024-04-20T02:52:01.655 INFO:tasks.rados.rados.0.smithi105.stdout:1357: tier_promote oid 42 2024-04-20T02:52:01.655 INFO:tasks.rados.rados.0.smithi105.stdout:1339: finishing write tid 2 to smithi10548823-3 2024-04-20T02:52:01.655 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:01.655 INFO:tasks.rados.rados.0.smithi105.stdout:1339: finishing write tid 3 to smithi10548823-3 2024-04-20T02:52:01.655 INFO:tasks.rados.rados.0.smithi105.stdout:1339: finishing write tid 4 to smithi10548823-3 2024-04-20T02:52:01.656 INFO:tasks.rados.rados.0.smithi105.stdout:1339: oid 3 updating version 0 to 282 2024-04-20T02:52:01.656 INFO:tasks.rados.rados.0.smithi105.stdout:1339: oid 3 version 282 is already newer than 280 2024-04-20T02:52:01.656 INFO:tasks.rados.rados.0.smithi105.stdout:1339: oid 3 version 282 is already newer than 281 2024-04-20T02:52:01.656 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 282 (ObjNum 250 snap 22 seq_num 250) dirty exists 2024-04-20T02:52:01.656 INFO:tasks.rados.rados.0.smithi105.stdout:1339: left oid 3 (ObjNum 250 snap 22 seq_num 250) 2024-04-20T02:52:01.656 INFO:tasks.rados.rados.0.smithi105.stdout:1339: done (15 left) 2024-04-20T02:52:01.656 INFO:tasks.rados.rados.0.smithi105.stdout:1358: delete oid 32 current snap is 22 2024-04-20T02:52:01.656 INFO:tasks.rados.rados.0.smithi105.stdout:1342: expect (ObjNum 235 snap 22 seq_num 235) 2024-04-20T02:52:01.679 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:01.680+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:01.743 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:01.743 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:01.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.799520027, 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:52:01.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:52:01.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.749 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.792041709000003, 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:01.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:52:01.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:52:01.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.787283810000005, 2024-04-20T02:52:01.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 67.363854876000005, 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.760 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:01.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:52:01.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:52:01.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:52:01.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:52:01.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 67.363817992999998, 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 67.363793360000003, 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.773 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:52:01.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:52:01.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:52:01.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 67.357299948999994, 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.778 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:52:01.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:52:01.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:52:01.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:52:01.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:52:01.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:52:01.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:52:01.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 67.351235927000005, 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.785 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.790 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 67.341174162000002, 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 67.340481557999993, 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:52:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:52:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:52:01.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:52:01.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 67.340469904000003, 2024-04-20T02:52:01.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.634405348000001, 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.807 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:52:01.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.615344632000003, 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.811 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:52:01.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.609398317999997, 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.817 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:01.822 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.608832876999998, 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.608806846, 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.828 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.832 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.6065940449999996, 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:52:01.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.6012445209999999, 2024-04-20T02:52:01.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:01.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.841 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.600774564, 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.842 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:52:01.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:52:01.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 5.6007520280000005, 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:01.846 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:01.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:52:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:52:01.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:01.850 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:01.851 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:01.852 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:01.852 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:01.852+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:01.889 INFO:tasks.rados.rados.0.smithi105.stdout:1343: finishing write tid 1 to smithi10548823-19 2024-04-20T02:52:01.889 INFO:tasks.rados.rados.0.smithi105.stdout:1343: finishing write tid 2 to smithi10548823-19 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:1343: finishing write tid 3 to smithi10548823-19 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:1343: finishing write tid 4 to smithi10548823-19 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:1343: oid 19 updating version 0 to 110 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:1343: oid 19 version 110 is already newer than 108 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:1343: oid 19 version 110 is already newer than 109 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 110 (ObjNum 251 snap 22 seq_num 251) dirty exists 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:1343: left oid 19 (ObjNum 251 snap 22 seq_num 251) 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:1345: finishing set_chunk to oid 20 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 92 (ObjNum 193 snap 19 seq_num 193) dirty exists 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:1346: finishing set_chunk to oid 6 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 96 (ObjNum 194 snap 19 seq_num 194) dirty exists 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:1347: finishing set_chunk to oid 24 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 273 (ObjNum 234 snap 22 seq_num 234) dirty exists 2024-04-20T02:52:01.890 INFO:tasks.rados.rados.0.smithi105.stdout:1348: finishing set_chunk to oid 25 2024-04-20T02:52:01.891 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 1980928 length: 550400 2024-04-20T02:52:01.891 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 0 (ObjNum 169 snap 14 seq_num 169) dirty exists 2024-04-20T02:52:01.891 INFO:tasks.rados.rados.0.smithi105.stdout:1351: finishing set_chunk to oid 8 2024-04-20T02:52:01.891 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 78 (ObjNum 243 snap 22 seq_num 243) dirty exists 2024-04-20T02:52:01.891 INFO:tasks.rados.rados.0.smithi105.stdout:1350: expect (ObjNum 237 snap 22 seq_num 237) 2024-04-20T02:52:01.953 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:01.953+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:1352: finishing set_chunk to oid 29 2024-04-20T02:52:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 20 (ObjNum 201 snap 19 seq_num 201) dirty exists 2024-04-20T02:52:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:1353: finishing write tid 1 to smithi10548823-17 2024-04-20T02:52:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:1353: finishing write tid 2 to smithi10548823-17 2024-04-20T02:52:02.017 INFO:tasks.rados.rados.0.smithi105.stdout:1353: finishing write tid 3 to smithi10548823-17 2024-04-20T02:52:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:1353: finishing write tid 4 to smithi10548823-17 2024-04-20T02:52:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:1353: oid 17 updating version 0 to 88 2024-04-20T02:52:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:1353: oid 17 version 88 is already newer than 87 2024-04-20T02:52:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:1353: oid 17 updating version 88 to 89 2024-04-20T02:52:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 89 (ObjNum 252 snap 22 seq_num 252) dirty exists 2024-04-20T02:52:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:1353: left oid 17 (ObjNum 252 snap 22 seq_num 252) 2024-04-20T02:52:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:1356: finishing set_chunk to oid 27 2024-04-20T02:52:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 152 (ObjNum 247 snap 22 seq_num 247) dirty exists 2024-04-20T02:52:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:1355: got (0) Success 2024-04-20T02:52:02.018 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 142 (ObjNum 151 snap 10 seq_num 151) dirty exists 2024-04-20T02:52:02.021 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:02.021 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:02.021 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:02.021 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:02.032 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:02.072 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:02.073+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:02.133 INFO:tasks.rados.rados.0.smithi105.stdout:1342: done (15 left) 2024-04-20T02:52:02.133 INFO:tasks.rados.rados.0.smithi105.stdout:1343: done (14 left) 2024-04-20T02:52:02.133 INFO:tasks.rados.rados.0.smithi105.stdout:1345: done (13 left) 2024-04-20T02:52:02.133 INFO:tasks.rados.rados.0.smithi105.stdout:1346: done (12 left) 2024-04-20T02:52:02.133 INFO:tasks.rados.rados.0.smithi105.stdout:1347: done (11 left) 2024-04-20T02:52:02.133 INFO:tasks.rados.rados.0.smithi105.stdout:1348: done (10 left) 2024-04-20T02:52:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:1350: done (9 left) 2024-04-20T02:52:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:1351: done (8 left) 2024-04-20T02:52:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:1352: done (7 left) 2024-04-20T02:52:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:1353: done (6 left) 2024-04-20T02:52:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:1355: done (5 left) 2024-04-20T02:52:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:1356: done (4 left) 2024-04-20T02:52:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:1358: done (3 left) 2024-04-20T02:52:02.134 INFO:tasks.rados.rados.0.smithi105.stdout:1359: set_chunk oid 8 target oid 2024-04-20T02:52:02.155 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:02.157+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:02.179 INFO:tasks.rados.rados.0.smithi105.stdout:1359: set_chunk oid 8 offset: 253440 length: 1088512 target oid 4ee26b2e652ebb80e59973090b78b3f028c78ed6866e1bc036e8386308b032a2 offset: 0 snap -1 2024-04-20T02:52:02.179 INFO:tasks.rados.rados.0.smithi105.stdout:1360: read oid 16 snap -1 2024-04-20T02:52:02.179 INFO:tasks.rados.rados.0.smithi105.stdout:1360: expect deleted 2024-04-20T02:52:02.179 INFO:tasks.rados.rados.0.smithi105.stdout:1361: set_chunk oid 50 target oid 2024-04-20T02:52:02.218 INFO:tasks.rados.rados.0.smithi105.stdout:1361: set_chunk oid 50 offset: 435200 length: 913408 target oid b2faa4f12d7cc0705e9572ec0b15996379add8e079fd680499ccc7725ce92280 offset: 0 snap -1 2024-04-20T02:52:02.218 INFO:tasks.rados.rados.0.smithi105.stdout:1362: read oid 19 snap -1 2024-04-20T02:52:02.218 INFO:tasks.rados.rados.0.smithi105.stdout:1362: expect (ObjNum 251 snap 22 seq_num 251) 2024-04-20T02:52:02.218 INFO:tasks.rados.rados.0.smithi105.stdout:1363: rollback oid 5 current snap is 22 2024-04-20T02:52:02.218 INFO:tasks.rados.rados.0.smithi105.stdout:1363: done (7 left) 2024-04-20T02:52:02.218 INFO:tasks.rados.rados.0.smithi105.stdout:1364: delete oid 47 current snap is 22 2024-04-20T02:52:02.218 INFO:tasks.rados.rados.0.smithi105.stdout:1349: expect (ObjNum 214 snap 20 seq_num 214) 2024-04-20T02:52:02.238 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:02.238 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:02.238 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:02.238 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:02.238 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:02.249 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:02.256 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:02.257+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:02.322 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:02.366 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:02.367+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:02.386 INFO:tasks.rados.rados.0.smithi105.stdout:1354: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:02.401 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.665947161000005, 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:02.402 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:02.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:52:02.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:52:02.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:52:02.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:52:02.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.634658099999996, 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:02.408 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:52:02.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:52:02.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:52:02.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:52:02.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:52:02.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:52:02.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:52:02.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:52:02.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:52:02.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:52:02.414 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.414 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.414 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.629967644000004, 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.415 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:52:02.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:52:02.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:52:02.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.623211174999994, 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:02.421 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:02.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:52:02.426 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.613395663999995, 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.427 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:52:02.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:52:02.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:02.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.612351731999993, 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:02.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:52:02.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:52:02.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:52:02.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:52:02.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:52:02.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:52:02.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:52:02.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.612330572000005, 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:02.439 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.443 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.404759991000006, 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:52:02.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:52:02.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.390565346000002, 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:52:02.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:52:02.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.384598088999994, 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.452 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:52:02.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:02.456 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.376192086000003, 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:52:02.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:52:02.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.272560045000006, 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.464 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.041963734999996, 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:52:02.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.468 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.015019585999994, 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:52:02.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:52:02.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.989661319000007, 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.961520875000005, 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:52:02.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.924152269999993, 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.481 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:52:02.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:52:02.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:52:02.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:52:02.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:52:02.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:02.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.923855347, 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:02.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.918772228999998, 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:52:02.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:52:02.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:52:02.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:52:02.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.495 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.891769670000002, 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:52:02.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:52:02.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:52:02.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:52:02.500 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.500 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.500 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.500 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:02.500 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:02.500 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:02.501 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:02.501 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:02.502 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:02.502+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:02.502 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:02.506 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:02.526 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:02.547 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:02.565 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:02.584 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:02.601 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:02.602+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:02.603 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:02.605 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:02.605 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:02.605 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:02.605 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:02.605 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:02.605 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:02.605 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:02.605 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:02.606 INFO:tasks.rados.rados.0.smithi105.stdout:1357: got (0) Success 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 61 (ObjNum 222 snap 21 seq_num 222) dirty exists 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1349: done (7 left) 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1354: done (6 left) 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1357: done (5 left) 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1364: done (4 left) 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1365: read oid 9 snap -1 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1365: expect (ObjNum 228 snap 22 seq_num 228) 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1366: rollback oid 18 current snap is 22 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1366: done (5 left) 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1367: read oid 40 snap -1 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1367: expect (ObjNum 248 snap 22 seq_num 248) 2024-04-20T02:52:02.607 INFO:tasks.rados.rados.0.smithi105.stdout:1368: set_chunk oid 21 target oid 2024-04-20T02:52:02.622 INFO:tasks.rados.rados.0.smithi105.stdout:1368: set_chunk oid 21 offset: 156672 length: 801280 target oid a9b5ee836bd841650e9d523a6c57fa799e5870c2755f6eefe9a9222251e9c6a8 offset: 0 snap -1 2024-04-20T02:52:02.622 INFO:tasks.rados.rados.0.smithi105.stdout:1369: write oid 5 current snap is 22 2024-04-20T02:52:02.622 INFO:tasks.rados.rados.0.smithi105.stdout:1359: finishing set_chunk to oid 8 2024-04-20T02:52:02.622 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 253440 length: 1088512 2024-04-20T02:52:02.622 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 243 snap 22 seq_num 243) dirty exists 2024-04-20T02:52:02.622 INFO:tasks.rados.rados.0.smithi105.stdout:1369: seq_num 255 ranges {685785=628144,1822320=463465} 2024-04-20T02:52:02.628 INFO:tasks.rados.rados.0.smithi105.stdout:1369: writing smithi10548823-5 from 685785 to 1313929 tid 1 2024-04-20T02:52:02.634 INFO:tasks.rados.rados.0.smithi105.stdout:1369: writing smithi10548823-5 from 1822320 to 2285785 tid 2 2024-04-20T02:52:02.634 INFO:tasks.rados.rados.0.smithi105.stdout:1359: done (7 left) 2024-04-20T02:52:02.634 INFO:tasks.rados.rados.0.smithi105.stdout:1370: read oid 24 snap -1 2024-04-20T02:52:02.634 INFO:tasks.rados.rados.0.smithi105.stdout:1370: expect (ObjNum 234 snap 22 seq_num 234) 2024-04-20T02:52:02.635 INFO:tasks.rados.rados.0.smithi105.stdout:1371: set_chunk oid 44 target oid 2024-04-20T02:52:02.638 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:02.639 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:02.652 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:02.665 INFO:tasks.rados.rados.0.smithi105.stdout:1371: set_chunk oid 44 offset: 2128896 length: 357888 target oid 19c55b9f6383aa8ecac7d4fd6466024b975baa7ca20de93149b6dace0bb9ae1f offset: 0 snap -1 2024-04-20T02:52:02.665 INFO:tasks.rados.rados.0.smithi105.stdout:1372: write (excl) oid 49 current snap is 22 2024-04-20T02:52:02.665 INFO:tasks.rados.rados.0.smithi105.stdout:1361: finishing set_chunk to oid 50 2024-04-20T02:52:02.665 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 84 (ObjNum 221 snap 21 seq_num 221) dirty exists 2024-04-20T02:52:02.665 INFO:tasks.rados.rados.0.smithi105.stdout:1362: expect (ObjNum 251 snap 22 seq_num 251) 2024-04-20T02:52:02.702 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:02.703+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:02.800 INFO:tasks.rados.rados.0.smithi105.stdout:1365: expect (ObjNum 228 snap 22 seq_num 228) 2024-04-20T02:52:02.802 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:02.804+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:02.821 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:02.821 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:02.831 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:02.905 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:02.904+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:02.961 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:02.961 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:02.971 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:03.012 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:03.013+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:03.013 INFO:tasks.rados.rados.0.smithi105.stdout:1367: expect (ObjNum 248 snap 22 seq_num 248) 2024-04-20T02:52:03.106 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:03.107+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:03.107 INFO:tasks.rados.rados.0.smithi105.stdout:1368: finishing set_chunk to oid 21 2024-04-20T02:52:03.107 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 21 set_chunk returned error code -95 offset: 156672 length: 801280 2024-04-20T02:52:03.107 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 0 (ObjNum 227 snap 22 seq_num 227) dirty exists 2024-04-20T02:52:03.107 INFO:tasks.rados.rados.0.smithi105.stdout:1370: expect (ObjNum 234 snap 22 seq_num 234) 2024-04-20T02:52:03.144 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:03.144 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:03.150 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:52:03.150 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:52:03.151 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-04-20T02:52:03.151 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-20T02:52:03.151 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-20T02:52:03.151 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-04-20T02:52:03.153 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:52:03.155 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:03.207 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:03.208+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:03.285 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:03.285 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:03.287 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:03.288 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-20T02:52:03.288 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:03.296 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:03.340 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:03.341+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:03.408 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:03.409+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:03.424 INFO:tasks.rados.rados.0.smithi105.stdout:1371: finishing set_chunk to oid 44 2024-04-20T02:52:03.424 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 72 (ObjNum 208 snap 19 seq_num 208) dirty exists 2024-04-20T02:52:03.424 INFO:tasks.rados.rados.0.smithi105.stdout:1369: finishing write tid 1 to smithi10548823-5 2024-04-20T02:52:03.424 INFO:tasks.rados.rados.0.smithi105.stdout:1369: finishing write tid 2 to smithi10548823-5 2024-04-20T02:52:03.424 INFO:tasks.rados.rados.0.smithi105.stdout:1372: seq_num 256 ranges {702592=703393,1880802=421790} 2024-04-20T02:52:03.425 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:03.425 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:03.431 INFO:tasks.rados.rados.0.smithi105.stdout:1372: writing smithi10548823-49 from 702592 to 1405985 tid 1 2024-04-20T02:52:03.436 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1372: writing smithi10548823-49 from 1880802 to 2302592 tid 2 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1360: done (9 left) 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1361: done (8 left) 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1362: done (7 left) 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1365: done (6 left) 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1367: done (5 left) 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1368: done (4 left) 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1370: done (3 left) 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1371: done (2 left) 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1373: write (excl) oid 8 current snap is 22 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1369: finishing write tid 3 to smithi10548823-5 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1369: finishing write tid 4 to smithi10548823-5 2024-04-20T02:52:03.438 INFO:tasks.rados.rados.0.smithi105.stdout:1369: oid 5 updating version 0 to 169 2024-04-20T02:52:03.439 INFO:tasks.rados.rados.0.smithi105.stdout:1369: oid 5 version 169 is already newer than 168 2024-04-20T02:52:03.439 INFO:tasks.rados.rados.0.smithi105.stdout:1369: oid 5 updating version 169 to 170 2024-04-20T02:52:03.439 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 170 (ObjNum 255 snap 22 seq_num 255) dirty exists 2024-04-20T02:52:03.439 INFO:tasks.rados.rados.0.smithi105.stdout:1369: left oid 5 (ObjNum 255 snap 22 seq_num 255) 2024-04-20T02:52:03.439 INFO:tasks.rados.rados.0.smithi105.stdout:1373: seq_num 257 ranges {719399=778642,1939284=380115} 2024-04-20T02:52:03.445 INFO:tasks.rados.rados.0.smithi105.stdout:1373: writing smithi10548823-8 from 719399 to 1498041 tid 1 2024-04-20T02:52:03.450 INFO:tasks.rados.rados.0.smithi105.stdout:1373: writing smithi10548823-8 from 1939284 to 2319399 tid 2 2024-04-20T02:52:03.451 INFO:tasks.rados.rados.0.smithi105.stdout:1369: done (2 left) 2024-04-20T02:52:03.451 INFO:tasks.rados.rados.0.smithi105.stdout:1374: read oid 7 snap -1 2024-04-20T02:52:03.451 INFO:tasks.rados.rados.0.smithi105.stdout:1374: expect (ObjNum 219 snap 21 seq_num 219) 2024-04-20T02:52:03.451 INFO:tasks.rados.rados.0.smithi105.stdout:1375: set_chunk oid 9 target oid 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1375: set_chunk oid 9 offset: 1305088 length: 665600 target oid 19925ed133031587cd9f7b0bb0f5a8be5fb6383fc480b65a819a1cdac08770c2 offset: 0 snap -1 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1376: set_chunk oid 32 target oid 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1376: set_chunk oid 32 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1377: read oid 27 snap -1 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1377: expect (ObjNum 247 snap 22 seq_num 247) 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1378: read oid 23 snap -1 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1378: expect (ObjNum 224 snap 22 seq_num 224) 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1379: read oid 45 snap -1 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1379: expect (ObjNum 220 snap 21 seq_num 220) 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1380: read oid 42 snap -1 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1380: expect (ObjNum 222 snap 21 seq_num 222) 2024-04-20T02:52:03.475 INFO:tasks.rados.rados.0.smithi105.stdout:1381: write (excl) oid 34 current snap is 22 2024-04-20T02:52:03.476 INFO:tasks.rados.rados.0.smithi105.stdout:1381: seq_num 258 ranges {736206=453891,1881413=454793} 2024-04-20T02:52:03.482 INFO:tasks.rados.rados.0.smithi105.stdout:1381: writing smithi10548823-34 from 736206 to 1190097 tid 1 2024-04-20T02:52:03.488 INFO:tasks.rados.rados.0.smithi105.stdout:1381: writing smithi10548823-34 from 1881413 to 2336206 tid 2 2024-04-20T02:52:03.488 INFO:tasks.rados.rados.0.smithi105.stdout:1382: tier_evict oid 6 2024-04-20T02:52:03.488 INFO:tasks.rados.rados.0.smithi105.stdout:1382: tier_evict oid 6 snap -1 2024-04-20T02:52:03.488 INFO:tasks.rados.rados.0.smithi105.stdout:1374: expect (ObjNum 219 snap 21 seq_num 219) 2024-04-20T02:52:03.509 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:03.509+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:03.609 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:03.611+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:03.610 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:03.611 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:03.621 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:03.710 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:03.711+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:1372: finishing write tid 1 to smithi10548823-49 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:1372: finishing write tid 2 to smithi10548823-49 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:1372: finishing write tid 3 to smithi10548823-49 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:1372: finishing write tid 4 to smithi10548823-49 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:1372: oid 49 updating version 0 to 164 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:1372: oid 49 updating version 164 to 165 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:1372: oid 49 updating version 165 to 166 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 166 (ObjNum 256 snap 22 seq_num 256) dirty exists 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:1372: left oid 49 (ObjNum 256 snap 22 seq_num 256) 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:1376: finishing set_chunk to oid 32 2024-04-20T02:52:03.769 INFO:tasks.rados.rados.0.smithi105.stdout:1376: got expected ENOENT (src dne) 2024-04-20T02:52:03.770 INFO:tasks.rados.rados.0.smithi105.stdout:1372: done (10 left) 2024-04-20T02:52:03.770 INFO:tasks.rados.rados.0.smithi105.stdout:1374: done (9 left) 2024-04-20T02:52:03.770 INFO:tasks.rados.rados.0.smithi105.stdout:1376: done (8 left) 2024-04-20T02:52:03.770 INFO:tasks.rados.rados.0.smithi105.stdout:1383: read oid 37 snap -1 2024-04-20T02:52:03.770 INFO:tasks.rados.rados.0.smithi105.stdout:1383: expect (ObjNum 184 snap 17 seq_num 184) 2024-04-20T02:52:03.770 INFO:tasks.rados.rados.0.smithi105.stdout:1384: set_chunk oid 50 target oid 2024-04-20T02:52:03.799 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:03.799 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:03.811 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:03.811 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:03.811+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:03.825 INFO:tasks.rados.rados.0.smithi105.stdout:1384: set_chunk oid 50 offset: 1166848 length: 2091520 target oid 4c679d080252ef302f97103426c94d78d7f95506df685b4f54a62c536479bf92 offset: 0 snap -1 2024-04-20T02:52:03.826 INFO:tasks.rados.rados.0.smithi105.stdout:1385: set_chunk oid 3 target oid 2024-04-20T02:52:03.849 INFO:tasks.rados.rados.0.smithi105.stdout:1385: set_chunk oid 3 offset: 1170432 length: 891904 target oid b1a01943fbf4e609db391c42950ac50f8172acfa8afacc503396427415816987 offset: 0 snap -1 2024-04-20T02:52:03.849 INFO:tasks.rados.rados.0.smithi105.stdout:1386: tier_flush oid 15 2024-04-20T02:52:03.849 INFO:tasks.rados.rados.0.smithi105.stdout:1386: tier_flush oid 15 snap -1 2024-04-20T02:52:03.849 INFO:tasks.rados.rados.0.smithi105.stdout:1377: expect (ObjNum 247 snap 22 seq_num 247) 2024-04-20T02:52:03.912 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:03.912+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:03.962 INFO:tasks.rados.rados.0.smithi105.stdout:1378: expect (ObjNum 224 snap 22 seq_num 224) 2024-04-20T02:52:03.969 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:03.969 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:03.979 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:04.012 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:04.013+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:04.113 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:04.114+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:04.146 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:52:04.157 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:04.157 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:04.168 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:04.178 INFO:tasks.rados.rados.0.smithi105.stdout:1373: finishing write tid 1 to smithi10548823-8 2024-04-20T02:52:04.178 INFO:tasks.rados.rados.0.smithi105.stdout:1373: finishing write tid 2 to smithi10548823-8 2024-04-20T02:52:04.178 INFO:tasks.rados.rados.0.smithi105.stdout:1373: finishing write tid 3 to smithi10548823-8 2024-04-20T02:52:04.178 INFO:tasks.rados.rados.0.smithi105.stdout:1373: finishing write tid 4 to smithi10548823-8 2024-04-20T02:52:04.178 INFO:tasks.rados.rados.0.smithi105.stdout:1373: oid 8 updating version 0 to 92 2024-04-20T02:52:04.178 INFO:tasks.rados.rados.0.smithi105.stdout:1373: oid 8 version 92 is already newer than 91 2024-04-20T02:52:04.178 INFO:tasks.rados.rados.0.smithi105.stdout:1373: oid 8 updating version 92 to 93 2024-04-20T02:52:04.178 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 93 (ObjNum 257 snap 22 seq_num 257) dirty exists 2024-04-20T02:52:04.179 INFO:tasks.rados.rados.0.smithi105.stdout:1373: left oid 8 (ObjNum 257 snap 22 seq_num 257) 2024-04-20T02:52:04.179 INFO:tasks.rados.rados.0.smithi105.stdout:1379: expect (ObjNum 220 snap 21 seq_num 220) 2024-04-20T02:52:04.221 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:04.223+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:04.306 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:04.307 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:04.316 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:04.316+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:04.317 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:04.355 INFO:tasks.rados.rados.0.smithi105.stdout:1382: got (22) Invalid argument 2024-04-20T02:52:04.356 INFO:tasks.rados.rados.0.smithi105.stdout:1380: expect (ObjNum 222 snap 21 seq_num 222) 2024-04-20T02:52:04.416 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:04.417+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:04.485 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:04.485 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:04.495 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:04.539 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:04.539+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:04.587 INFO:tasks.rados.rados.0.smithi105.stdout:1375: finishing set_chunk to oid 9 2024-04-20T02:52:04.587 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 232 (ObjNum 228 snap 22 seq_num 228) dirty exists 2024-04-20T02:52:04.587 INFO:tasks.rados.rados.0.smithi105.stdout:1381: finishing write tid 1 to smithi10548823-34 2024-04-20T02:52:04.587 INFO:tasks.rados.rados.0.smithi105.stdout:1381: finishing write tid 2 to smithi10548823-34 2024-04-20T02:52:04.587 INFO:tasks.rados.rados.0.smithi105.stdout:1373: done (11 left) 2024-04-20T02:52:04.587 INFO:tasks.rados.rados.0.smithi105.stdout:1375: done (10 left) 2024-04-20T02:52:04.587 INFO:tasks.rados.rados.0.smithi105.stdout:1377: done (9 left) 2024-04-20T02:52:04.587 INFO:tasks.rados.rados.0.smithi105.stdout:1378: done (8 left) 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1379: done (7 left) 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1380: done (6 left) 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1382: done (5 left) 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1387: read oid 47 snap -1 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1387: expect deleted 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1388: read oid 41 snap -1 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1388: expect deleted 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1389: read oid 26 snap -1 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1389: expect (ObjNum 223 snap 22 seq_num 223) 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1390: delete oid 9 current snap is 22 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1381: finishing write tid 3 to smithi10548823-34 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1381: finishing write tid 4 to smithi10548823-34 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1381: oid 34 updating version 0 to 241 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1381: oid 34 version 241 is already newer than 239 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1381: oid 34 version 241 is already newer than 240 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 241 (ObjNum 258 snap 22 seq_num 258) dirty exists 2024-04-20T02:52:04.588 INFO:tasks.rados.rados.0.smithi105.stdout:1381: left oid 34 (ObjNum 258 snap 22 seq_num 258) 2024-04-20T02:52:04.589 INFO:tasks.rados.rados.0.smithi105.stdout:1383: expect (ObjNum 184 snap 17 seq_num 184) 2024-04-20T02:52:04.619 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:04.619+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:04.624 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:04.627 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:04.627 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:04.639 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:04.723 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:04.723+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:04.773 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:04.792 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:04.811 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:04.812 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:04.812 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:04.812 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:04.812 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:04.812 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:04.819 INFO:tasks.rados.rados.0.smithi105.stdout:1384: finishing set_chunk to oid 50 2024-04-20T02:52:04.819 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 50 set_chunk returned error code -95 offset: 1166848 length: 2091520 2024-04-20T02:52:04.819 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 0 (ObjNum 221 snap 21 seq_num 221) dirty exists 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1385: finishing set_chunk to oid 3 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 282 (ObjNum 250 snap 22 seq_num 250) dirty exists 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1386: got (0) Success 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 0 (ObjNum 231 snap 22 seq_num 231) dirty exists 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1381: done (8 left) 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1383: done (7 left) 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1384: done (6 left) 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1385: done (5 left) 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1386: done (4 left) 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1390: done (3 left) 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1391: read oid 48 snap -1 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1391: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1392: read oid 4 snap -1 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1392: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1393: rollback oid 20 current snap is 22 2024-04-20T02:52:04.820 INFO:tasks.rados.rados.0.smithi105.stdout:1393: done (5 left) 2024-04-20T02:52:04.821 INFO:tasks.rados.rados.0.smithi105.stdout:1394: read oid 46 snap -1 2024-04-20T02:52:04.821 INFO:tasks.rados.rados.0.smithi105.stdout:1394: expect (ObjNum 232 snap 22 seq_num 232) 2024-04-20T02:52:04.821 INFO:tasks.rados.rados.0.smithi105.stdout:1395: set_chunk oid 5 target oid 2024-04-20T02:52:04.823 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:04.822+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:04.826 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:04.829 INFO:tasks.rados.rados.0.smithi105.stdout:1395: set_chunk oid 5 offset: 174080 length: 453632 target oid d356ec7312dc0168af472d1d64ca0e79a5b614113383e963cf4f3b8b28e9ec9d offset: 0 snap -1 2024-04-20T02:52:04.829 INFO:tasks.rados.rados.0.smithi105.stdout:1396: rollback oid 12 current snap is 22 2024-04-20T02:52:04.829 INFO:tasks.rados.rados.0.smithi105.stdout:1396: done (7 left) 2024-04-20T02:52:04.829 INFO:tasks.rados.rados.0.smithi105.stdout:1397: write oid 49 current snap is 22 2024-04-20T02:52:04.829 INFO:tasks.rados.rados.0.smithi105.stdout:1397: seq_num 260 ranges {769820=720742,2114730=255090} 2024-04-20T02:52:04.837 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:04.837 INFO:tasks.rados.rados.0.smithi105.stdout:1397: writing smithi10548823-49 from 769820 to 1490562 tid 1 2024-04-20T02:52:04.842 INFO:tasks.rados.rados.0.smithi105.stdout:1397: writing smithi10548823-49 from 2114730 to 2369820 tid 2 2024-04-20T02:52:04.843 INFO:tasks.rados.rados.0.smithi105.stdout:1398: snap_create 2024-04-20T02:52:04.843 INFO:tasks.rados.rados.0.smithi105.stdout:1389: expect (ObjNum 223 snap 22 seq_num 223) 2024-04-20T02:52:04.870 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:04.890 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:04.910 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:04.922 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:04.923+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:04.930 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:04.931 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:04.931 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:04.931 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:04.931 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:04.931 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:04.931 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:04.931 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:04.932 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:04.966 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:04.966 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:04.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:04.966 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 74.020499916999995, 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041399200999999997, 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945228+0000", 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.945227+0000", 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947429+0000", 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022028989999999999 2024-04-20T02:52:04.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947448+0000", 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.821e-05 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947460+0000", 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2373e-05 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947511+0000", 2024-04-20T02:52:04.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0844999999999999e-05 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.947531+0000", 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0017000000000002e-05 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986580+0000", 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039049213999999999 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.986627+0000", 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7070000000000002e-05 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1145 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'30, mlcod=34'26)", 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 74.001769218000007, 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026752594000000001, 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:04.971 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963959+0000", 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.963958+0000", 2024-04-20T02:52:04.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964662+0000", 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070467299999999995 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964678+0000", 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5597999999999999e-05 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964687+0000", 2024-04-20T02:52:04.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2979999999999997e-06 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968184+0000", 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0034972250000000001 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.968211+0000", 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6829999999999999e-05 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:04.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990640+0000", 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022429317000000001 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.990711+0000", 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0794e-05 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1462 3.0 e44/21 3:29b349f1:::ea1112aea5e7be379214f0429a6e898cf4f4a05bdbb29b6cd15764b4776418b1:head v 44'85, mlcod=42'83)", 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:52:04.975 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 63.787938558999997, 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034060275000000001, 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.177789+0000", 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178932+0000", 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0011426069999999999 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178944+0000", 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.287e-05 2024-04-20T02:52:04.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.178962+0000", 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7669999999999999e-05 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179013+0000", 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1295000000000003e-05 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.179038+0000", 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4859999999999999e-05 2024-04-20T02:52:04.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211813+0000", 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032775190000000003 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:01.211850+0000", 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6216e-05 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1629 3.0 e51/21 3:088cc2f3:::a1744cc895df5e9d3e624b281fcb9edc57f3f342d1f108d4c38dc778f219a41a:head v 51'93, mlcod=45'91)", 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.162639222000003, 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035323188999999998, 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:04.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803089+0000", 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.803088+0000", 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804187+0000", 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00109891 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804201+0000", 2024-04-20T02:52:04.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3772e-05 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804213+0000", 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2335e-05 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804285+0000", 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1271000000000003e-05 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.804306+0000", 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1359999999999999e-05 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838377+0000", 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034070994 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:08.838412+0000", 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4790999999999999e-05 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1653 4.6 e51/22 4:697da67d:::smithi10548823-48:head v 51'41, mlcod=45'37)", 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.540900114000003, 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037653640000000002, 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424828+0000", 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.424827+0000", 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425066+0000", 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023957399999999999 2024-04-20T02:52:04.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425117+0000", 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1082999999999999e-05 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425146+0000", 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8129000000000001e-05 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425217+0000", 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1402000000000005e-05 2024-04-20T02:52:04.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.425278+0000", 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1498e-05 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462456+0000", 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037177106000000001 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.462481+0000", 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5851e-05 2024-04-20T02:52:04.987 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1786 4.d e56/22 4:b376c20e:::smithi10548823-49:head v 56'80, mlcod=55'77)", 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.153808577, 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028136330000000001, 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:04.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.811919+0000", 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812644+0000", 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072510100000000002 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812656+0000", 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1688999999999999e-05 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812663+0000", 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812707+0000", 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4573999999999997e-05 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.812722+0000", 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5319e-05 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840021+0000", 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027298111999999999 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:04.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:13.840056+0000", 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5086000000000003e-05 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1799 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'53, mlcod=54'49)", 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.891479556, 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050527214000000001, 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074248+0000", 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:04.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074614+0000", 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036631499999999998 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074624+0000", 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8330000000000006e-06 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:04.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074632+0000", 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3599999999999998e-06 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074683+0000", 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1137999999999997e-05 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.074704+0000", 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1616000000000001e-05 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124714+0000", 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050009642 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124776+0000", 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1581000000000006e-05 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.996 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1806 4.d e56/22 4:b9e59c86:::smithi10548823-2:head v 56'86, mlcod=56'82)", 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.876983459999998, 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046184680999999998, 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088744+0000", 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.088743+0000", 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090327+0000", 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001584071 2024-04-20T02:52:04.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090343+0000", 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5193e-05 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090354+0000", 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1336e-05 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:04.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090411+0000", 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6589999999999999e-05 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.090431+0000", 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0579000000000001e-05 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134883+0000", 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044451877000000001 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.134929+0000", 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6131000000000003e-05 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1800 4.6 e56/22 4:697da67d:::smithi10548823-48:head v 56'54, mlcod=54'49)", 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.864577386000001, 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027029681999999999, 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.001 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101151+0000", 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101150+0000", 2024-04-20T02:52:05.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101284+0000", 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000134688 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101295+0000", 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101305+0000", 2024-04-20T02:52:05.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0930999999999999e-05 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101347+0000", 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1522e-05 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.101369+0000", 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2027000000000001e-05 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128138+0000", 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026768626 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.128180+0000", 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2568999999999998e-05 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2038 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'159, mlcod=0'0)", 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 40.384107987, 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044306238999999997, 2024-04-20T02:52:05.005 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581620+0000", 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581964+0000", 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034420099999999998 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581980+0000", 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6578999999999998e-05 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.581991+0000", 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0949e-05 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582734+0000", 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074277299999999996 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.582761+0000", 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6627000000000001e-05 2024-04-20T02:52:05.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625882+0000", 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043121617000000001 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.625926+0000", 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3838999999999998e-05 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2039 4.19 e67/65 4:9c87d36d:::smithi10548823-24:head v 67'160, mlcod=0'0)", 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 40.363709901999997, 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027318719000000002, 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602018+0000", 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602017+0000", 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602364+0000", 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000347139 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602375+0000", 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.128e-05 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.602384+0000", 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2600000000000005e-06 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605223+0000", 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0028393400000000001 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.605245+0000", 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2078000000000001e-05 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629297+0000", 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024051605 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.629337+0000", 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0126000000000002e-05 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.013 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2142 4.6 e70/60 4:677c62b0:::smithi10548823-23:head v 70'65, mlcod=66'61)", 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.148882505000003, 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033365672999999998, 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816845+0000", 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816981+0000", 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135952 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.816994+0000", 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3405e-05 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817003+0000", 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4319999999999999e-06 2024-04-20T02:52:05.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817055+0000", 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1842999999999998e-05 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.817076+0000", 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1375000000000001e-05 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850179+0000", 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033103051000000001 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:27.850211+0000", 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1896000000000001e-05 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:52:05.018 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 30.824347889999999, 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039421115999999999, 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141380+0000", 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141379+0000", 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142797+0000", 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001418702 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142816+0000", 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8907999999999999e-05 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142825+0000", 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1020000000000003e-06 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142872+0000", 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6168999999999998e-05 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142891+0000", 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9561000000000001e-05 2024-04-20T02:52:05.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180704+0000", 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037812643 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.180801+0000", 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7361999999999994e-05 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2280 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'105, mlcod=73'100)", 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 30.803666613000001, 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029657706999999998, 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162061+0000", 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162060+0000", 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162455+0000", 2024-04-20T02:52:05.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039569799999999999 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162472+0000", 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6820000000000002e-05 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.162485+0000", 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2388999999999999e-05 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163758+0000", 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001273906 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.163784+0000", 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5352000000000001e-05 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191663+0000", 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027879583999999999 2024-04-20T02:52:05.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.191719+0000", 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640000000000003e-05 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2559 3.0 e91/21 3:0870df8d:::348e4a2a55868b09db2e6d173951b4676cfef7376c5a81f7a82bb8f7ccc4ada5:head v 91'132, mlcod=91'130)", 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 16.496910851999999, 2024-04-20T02:52:05.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032176705999999999, 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468817+0000", 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.468815+0000", 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469831+0000", 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001015076 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469853+0000", 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2713000000000001e-05 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469865+0000", 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1904e-05 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469910+0000", 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5114999999999997e-05 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469934+0000", 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4133000000000001e-05 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.500951+0000", 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031016851000000002 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.500994+0000", 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2496000000000001e-05 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.031 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2788 4.14 e98/97 4:2e88ea27:::smithi10548823-12:head v 98'31, mlcod=0'0)", 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:56.144245+0000", 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 8.8214831880000002, 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038004836, 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144245+0000", 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144245+0000", 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144244+0000", 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144489+0000", 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024499599999999998 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144512+0000", 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2989e-05 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144521+0000", 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2769999999999994e-06 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144580+0000", 2024-04-20T02:52:05.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9119000000000001e-05 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.144612+0000", 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1962000000000003e-05 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.182209+0000", 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037597034000000001 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:56.182250+0000", 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0376000000000001e-05 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2930 3.0 e104/21 3:00ea3e76:::4ee26b2e652ebb80e59973090b78b3f028c78ed6866e1bc036e8386308b032a2:head v 104'154, mlcod=101'152)", 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:02.145968+0000", 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.8197599699999998, 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029251332000000001, 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.036 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.145968+0000", 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.145968+0000", 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.145968+0000", 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.146555+0000", 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00058734399999999995 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.146565+0000", 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0586e-05 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.146573+0000", 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2570000000000001e-06 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.146636+0000", 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3510999999999996e-05 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.146660+0000", 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4020999999999999e-05 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.175190+0000", 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028529327 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.175219+0000", 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9674000000000001e-05 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.040 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDPGPush(4.14 104/103 [PushOp(4:2e88ea27:::smithi10548823-12:2c, version: 98'30, data_included: [1580315~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:2e88ea27:::smithi10548823-12:2c@98'30, size: 2806769, copy_subset: [0~2806769], clone_subset: {}, snapset: 2c=[]:{2c=[2c]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2628891, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1580315, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:02.181525+0000", 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.7842033470000001, 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029565887999999999, 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.181525+0000", 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.181525+0000", 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.181524+0000", 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.181774+0000", 2024-04-20T02:52:05.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025092799999999998 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.181792+0000", 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8025000000000001e-05 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.181810+0000", 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7235999999999999e-05 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.181886+0000", 2024-04-20T02:52:05.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6100999999999995e-05 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.181902+0000", 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6027000000000001e-05 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.211090+0000", 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029188591 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "MOSDPGPush(4.16 104/103 [PushOp(4:697da67d:::smithi10548823-48:head, version: 97'80, data_included: [1517513~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:697da67d:::smithi10548823-48:head@97'80, size: 3277365, copy_subset: [0~3277365], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2566089, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1517513, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:02.233526+0000", 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.732202332, 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031254601999999999, 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.233526+0000", 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.233526+0000", 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.233525+0000", 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.234047+0000", 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052254099999999996 2024-04-20T02:52:05.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.234065+0000", 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8091999999999999e-05 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.234083+0000", 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7810999999999999e-05 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.234167+0000", 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4207000000000002e-05 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.234184+0000", 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6143000000000001e-05 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:02.264780+0000", 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030596600000000002 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.048 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:3025 4.d e106/22 4:b376c20e:::smithi10548823-49:head v 106'167, mlcod=105'165)", 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:04.840897+0000", 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.124831188, 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034603938000000001, 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:04.840897+0000", 2024-04-20T02:52:05.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:04.840897+0000", 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:04.840896+0000", 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:04.842270+0000", 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001373911 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:04.842286+0000", 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6002e-05 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:04.842296+0000", 2024-04-20T02:52:05.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1980000000000007e-06 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:04.842368+0000", 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2324999999999996e-05 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:04.842395+0000", 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7161999999999998e-05 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:04.875460+0000", 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033064860000000001 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:04.875501+0000", 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0747999999999997e-05 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.053 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:05.054 INFO:tasks.rados.rados.0.smithi105.stdout:1391: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:05.055 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:05.055 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:05.055+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:05.155 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:05.155+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:05.185 INFO:tasks.rados.rados.0.smithi105.stdout:1394: expect (ObjNum 232 snap 22 seq_num 232) 2024-04-20T02:52:05.228 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:05.229 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:05.229 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:05.229 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:05.239 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:05.280 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:05.280+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:05.358 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:05.359+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:05.404 INFO:tasks.rados.rados.0.smithi105.stdout:1392: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:05.408 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:05.409 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:05.409 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:05.409 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:05.409 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:05.419 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:05.459 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:05.459+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:05.553 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:05.553 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:05.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.008677938000005, 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:52:05.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:52:05.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:52:05.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:52:05.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:52:05.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:52:05.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:52:05.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:05.560 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.002456863999996, 2024-04-20T02:52:05.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:52:05.561 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.561 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.561 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.561 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.561 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.561 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:52:05.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.561 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:52:05.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:52:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:52:05.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.566 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 78.994056659999998, 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:05.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:05.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:52:05.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:52:05.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:52:05.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:52:05.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:52:05.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:52:05.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:05.574 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 78.993714014999995, 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:05.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:52:05.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:52:05.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:52:05.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 71.187392012000004, 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.579 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:05.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:52:05.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:52:05.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:52:05.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 71.180129050000005, 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.585 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:52:05.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:52:05.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:52:05.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:52:05.590 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 71.174186520999996, 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:52:05.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:52:05.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.596 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 71.173983528999997, 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.597 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:52:05.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:52:05.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:52:05.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 71.173960601999994, 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:05.607 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 71.163876868000003, 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:52:05.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.611 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 71.146991459999995, 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:05.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:52:05.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 71.130715018999993, 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.619 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.106159546999997, 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.620 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:52:05.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.625 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 40.791843771000003, 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:52:05.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2545 4.9 e91/81 4:95e19a88:::smithi10548823-40:head v 91'253, mlcod=91'251)", 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.137206656, 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054757404000000003, 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:05.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415210+0000", 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416004+0000", 2024-04-20T02:52:05.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079381199999999997 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416022+0000", 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7969e-05 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416031+0000", 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9279999999999999e-06 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416100+0000", 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8891000000000002e-05 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416128+0000", 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8203000000000001e-05 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469912+0000", 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053783794000000003 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469968+0000", 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6474000000000002e-05 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.204177849000001, 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.636 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2700 4.0 4:007b2a87:::smithi10548823-39:head [write 1967200~714747] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.195687655, 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061846287, 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.642 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2700 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:05.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356728+0000", 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356879+0000", 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000150958 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356894+0000", 2024-04-20T02:52:05.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5132e-05 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356906+0000", 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2071e-05 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366666+0000", 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0097595289999999994 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366774+0000", 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108527 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366956+0000", 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181359 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390515+0000", 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023559902000000001 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406790+0000", 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016274555999999999 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418488+0000", 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011697591 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418529+0000", 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1545000000000003e-05 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418576+0000", 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6903000000000001e-05 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2701 4.0 4:007b2a87:::smithi10548823-39:head [write 3421983~426820] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.188511663, 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054799176999999998, 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.648 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2701 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363904+0000", 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364154+0000", 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000250179 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364171+0000", 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6748e-05 2024-04-20T02:52:05.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364182+0000", 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1371e-05 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390552+0000", 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026369607 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390652+0000", 2024-04-20T02:52:05.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010059499999999999 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390857+0000", 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020445199999999999 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418224+0000", 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027367316999999999 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418282+0000", 2024-04-20T02:52:05.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8279000000000002e-05 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418618+0000", 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000335967 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418661+0000", 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2614999999999998e-05 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418705+0000", 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3993999999999999e-05 2024-04-20T02:52:05.653 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2702 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (57) in=7b,truncate 3848803] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.188225609, 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058669165000000002, 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2702 2024-04-20T02:52:05.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364191+0000", 2024-04-20T02:52:05.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364196+0000", 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4490000000000001e-06 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364201+0000", 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5040000000000002e-06 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364206+0000", 2024-04-20T02:52:05.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.405895+0000", 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041689047 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406021+0000", 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000125561 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406541+0000", 2024-04-20T02:52:05.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052000699999999996 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418451+0000", 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011910041999999999 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418747+0000", 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029622399999999999 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422704+0000", 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039571859999999997 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422781+0000", 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.661e-05 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422861+0000", 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9987999999999994e-05 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2703 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.188206557000001, 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058940278999999998, 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2703 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:05.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364213+0000", 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4830000000000002e-06 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364220+0000", 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.007e-06 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364225+0000", 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0350000000000001e-06 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406820+0000", 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042595135999999999 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406923+0000", 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102846 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422897+0000", 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015974202999999999 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423015+0000", 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117581 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423151+0000", 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013635899999999999 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:05.664 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:05.666 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:05.670 INFO:tasks.rados.rados.0.smithi105.stdout:1395: finishing set_chunk to oid 5 2024-04-20T02:52:05.670 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 170 (ObjNum 255 snap 22 seq_num 255) dirty exists 2024-04-20T02:52:05.670 INFO:tasks.rados.rados.0.smithi105.stdout:1397: finishing write tid 1 to smithi10548823-49 2024-04-20T02:52:05.670 INFO:tasks.rados.rados.0.smithi105.stdout:1397: finishing write tid 2 to smithi10548823-49 2024-04-20T02:52:05.670 INFO:tasks.rados.rados.0.smithi105.stdout:1397: finishing write tid 3 to smithi10548823-49 2024-04-20T02:52:05.670 INFO:tasks.rados.rados.0.smithi105.stdout:1397: finishing write tid 4 to smithi10548823-49 2024-04-20T02:52:05.670 INFO:tasks.rados.rados.0.smithi105.stdout:1397: oid 49 updating version 0 to 167 2024-04-20T02:52:05.670 INFO:tasks.rados.rados.0.smithi105.stdout:1397: oid 49 updating version 167 to 169 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1397: oid 49 version 169 is already newer than 168 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 169 (ObjNum 260 snap 22 seq_num 260) dirty exists 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1397: left oid 49 (ObjNum 260 snap 22 seq_num 260) 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1387: done (8 left) 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1388: done (7 left) 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1389: done (6 left) 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1391: done (5 left) 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1392: done (4 left) 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1394: done (3 left) 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1395: done (2 left) 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1397: done (1 left) 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1398: done (0 left) 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1399: read oid 22 snap 22 2024-04-20T02:52:05.671 INFO:tasks.rados.rados.0.smithi105.stdout:1399: expect (ObjNum 246 snap 22 seq_num 246) 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1400: read oid 16 snap -1 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1400: expect deleted 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1401: read oid 30 snap -1 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1401: expect (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1402: delete oid 17 current snap is 23 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1400: done (3 left) 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1402: done (2 left) 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1403: read oid 16 snap 22 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1403: expect deleted 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1404: read oid 3 snap -1 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1404: expect (ObjNum 250 snap 22 seq_num 250) 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1405: read oid 43 snap -1 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1405: expect (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1406: write oid 5 current snap is 23 2024-04-20T02:52:05.672 INFO:tasks.rados.rados.0.smithi105.stdout:1406: seq_num 262 ranges {403434=471240,1315341=400756} 2024-04-20T02:52:05.673 INFO:tasks.rados.rados.0.smithi105.stdout:1406: writing smithi10548823-5 from 403434 to 874674 tid 1 2024-04-20T02:52:05.673 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:05.666+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:05.674 INFO:tasks.rados.rados.0.smithi105.stdout:1406: writing smithi10548823-5 from 1315341 to 1716097 tid 2 2024-04-20T02:52:05.674 INFO:tasks.rados.rados.0.smithi105.stdout:1407: set_chunk oid 18 target oid 2024-04-20T02:52:05.692 INFO:tasks.rados.rados.0.smithi105.stdout:1407: set_chunk oid 18 offset: 2129920 length: 875008 target oid d750e94eeaf729bced4a71a50e819706650f866c9bf26d25649e8ac6a073954c offset: 0 snap -1 2024-04-20T02:52:05.692 INFO:tasks.rados.rados.0.smithi105.stdout:1408: tier_evict oid 39 2024-04-20T02:52:05.692 INFO:tasks.rados.rados.0.smithi105.stdout:1408: tier_evict oid 39 snap -1 2024-04-20T02:52:05.692 INFO:tasks.rados.rados.0.smithi105.stdout:1399: expect (ObjNum 246 snap 22 seq_num 246) 2024-04-20T02:52:05.767 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:05.767+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:05.807 INFO:tasks.rados.rados.0.smithi105.stdout:1401: expect (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:52:05.815 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:05.815 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:05.815 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:05.815 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:05.826 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:05.869 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:05.870+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:05.955 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:05.955 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:05.955 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:05.955 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:05.955 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:05.965 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:06.007 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:06.007+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:06.069 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:06.070+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:06.140 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:06.140 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:06.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 75.196697809, 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.141 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:06.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:52:06.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:52:06.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:52:06.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 75.189219491000003, 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:06.152 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 75.184461592000005, 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:06.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.157 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.761032658000005, 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.158 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:52:06.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:06.164 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.760995774999998, 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:52:06.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:52:06.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.760971142000002, 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:52:06.171 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.754477730999994, 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.176 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:52:06.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:52:06.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:52:06.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:52:06.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:52:06.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.182 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.182 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:52:06.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:52:06.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.182 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.748413709000005, 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.183 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:52:06.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:52:06.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.188 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.738351944000001, 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.189 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:52:06.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.194 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.737659339999993, 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:06.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:52:06.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:52:06.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.737647686000003, 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.205 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.031583130000001, 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.209 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.012522414000003, 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:52:06.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:52:06.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.006576100000004, 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.216 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:52:06.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:06.221 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.006010658999998, 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:06.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:52:06.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:52:06.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:06.227 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.005984627999993, 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:52:06.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:06.232 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 10.003771827, 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:52:06.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.236 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.9984223029999999, 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.237 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:52:06.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:52:06.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.9979523459999999, 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:52:06.241 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.245 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.9979298100000005, 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.246 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:52:06.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:06.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.250 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.250 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.250 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:06.250 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:52:06.251 INFO:tasks.rados.rados.0.smithi105.stdout:1404: expect (ObjNum 250 snap 22 seq_num 250) 2024-04-20T02:52:06.252 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:06.253 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:06.252+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:06.261 INFO:tasks.rados.rados.0.smithi105.stdout:1405: expect (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:52:06.352 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:06.353+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:06.423 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:06.423 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:06.423 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:06.423 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:06.432 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:06.473 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:06.474+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:06.554 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:06.555+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:06.569 INFO:tasks.rados.rados.0.smithi105.stdout:1406: finishing write tid 1 to smithi10548823-5 2024-04-20T02:52:06.569 INFO:tasks.rados.rados.0.smithi105.stdout:1406: finishing write tid 2 to smithi10548823-5 2024-04-20T02:52:06.569 INFO:tasks.rados.rados.0.smithi105.stdout:1399: done (7 left) 2024-04-20T02:52:06.569 INFO:tasks.rados.rados.0.smithi105.stdout:1401: done (6 left) 2024-04-20T02:52:06.569 INFO:tasks.rados.rados.0.smithi105.stdout:1403: done (5 left) 2024-04-20T02:52:06.569 INFO:tasks.rados.rados.0.smithi105.stdout:1404: done (4 left) 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1405: done (3 left) 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1409: read oid 42 snap 22 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1409: expect (ObjNum 222 snap 21 seq_num 222) 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1410: snap_create 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1408: got (22) Invalid argument 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1406: finishing write tid 3 to smithi10548823-5 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1406: finishing write tid 4 to smithi10548823-5 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1406: oid 5 updating version 0 to 173 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1406: oid 5 updating version 173 to 175 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1406: oid 5 version 175 is already newer than 174 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 175 (ObjNum 262 snap 23 seq_num 262) dirty exists 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1406: left oid 5 (ObjNum 262 snap 23 seq_num 262) 2024-04-20T02:52:06.570 INFO:tasks.rados.rados.0.smithi105.stdout:1407: finishing set_chunk to oid 18 2024-04-20T02:52:06.571 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 93 (ObjNum 237 snap 22 seq_num 237) dirty exists 2024-04-20T02:52:06.578 INFO:tasks.rados.rados.0.smithi105.stdout:1409: expect (ObjNum 222 snap 21 seq_num 222) 2024-04-20T02:52:06.603 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:06.603 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:06.604 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:06.604 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:06.604 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:06.614 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:06.655 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:06.655+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:06.756 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:06.756+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:06.785 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:06.785 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:06.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:06.785 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:06.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.785 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:06.785 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:06.785 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 85.049110908000003, 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:06.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:52:06.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:52:06.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.791 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 85.017821846999993, 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:06.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:52:06.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 85.013131391000002, 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:52:06.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:52:06.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.805 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:06.805 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:06.805 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 85.006374922000006, 2024-04-20T02:52:06.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:52:06.805 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.805 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.805 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.805 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:06.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:52:06.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:52:06.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:52:06.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.996559411000007, 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:06.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:06.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:52:06.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:52:06.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.995515479000005, 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:06.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.823 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.995494319000002, 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:06.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:06.828 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.787923738000003, 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.832 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.773729093, 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:52:06.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:52:06.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:52:06.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.767761836000005, 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:52:06.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.759355833000001, 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:52:06.841 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:06.845 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.655723792000003, 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:52:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:52:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:52:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:52:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.849 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.425127481999994, 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:06.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:52:06.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:52:06.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:52:06.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.398183333000006, 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.372825066000004, 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.344684622000003, 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.862 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:52:06.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:52:06.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:52:06.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.307316017000005, 2024-04-20T02:52:06.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:52:06.870 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.307019093999997, 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:06.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:52:06.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:52:06.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.301935975999996, 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.878 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.274933417, 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.882 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.883 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.883 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:06.883 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:06.883 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:06.884 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:06.885 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:06.885+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:06.944 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:06.985 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:06.985+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:07.054 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:07.054 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:07.054 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:07.054 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:07.064 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:07.085 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:07.086+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:07.089 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:07.108 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:07.127 INFO:tasks.rados.rados.0.smithi105.stdout:1406: done (4 left) 2024-04-20T02:52:07.127 INFO:tasks.rados.rados.0.smithi105.stdout:1407: done (3 left) 2024-04-20T02:52:07.127 INFO:tasks.rados.rados.0.smithi105.stdout:1408: done (2 left) 2024-04-20T02:52:07.127 INFO:tasks.rados.rados.0.smithi105.stdout:1409: done (1 left) 2024-04-20T02:52:07.127 INFO:tasks.rados.rados.0.smithi105.stdout:1410: done (0 left) 2024-04-20T02:52:07.127 INFO:tasks.rados.rados.0.smithi105.stdout:1411: read oid 7 snap -1 2024-04-20T02:52:07.127 INFO:tasks.rados.rados.0.smithi105.stdout:1411: expect (ObjNum 219 snap 21 seq_num 219) 2024-04-20T02:52:07.127 INFO:tasks.rados.rados.0.smithi105.stdout:1412: tier_promote oid 3 2024-04-20T02:52:07.127 INFO:tasks.rados.rados.0.smithi105.stdout:1413: write (excl) oid 37 current snap is 24 2024-04-20T02:52:07.127 INFO:tasks.rados.rados.0.smithi105.stdout:1413: seq_num 263 ranges {420241=546489,1373823=544414} 2024-04-20T02:52:07.129 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:07.131 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:52:07.132 INFO:tasks.rados.rados.0.smithi105.stdout:1413: writing smithi10548823-37 from 420241 to 966730 tid 1 2024-04-20T02:52:07.139 INFO:tasks.rados.rados.0.smithi105.stdout:1413: writing smithi10548823-37 from 1373823 to 1918237 tid 2 2024-04-20T02:52:07.139 INFO:tasks.rados.rados.0.smithi105.stdout:1414: read oid 34 snap -1 2024-04-20T02:52:07.140 INFO:tasks.rados.rados.0.smithi105.stdout:1414: expect (ObjNum 258 snap 22 seq_num 258) 2024-04-20T02:52:07.140 INFO:tasks.rados.rados.0.smithi105.stdout:1415: delete oid 21 current snap is 24 2024-04-20T02:52:07.140 INFO:tasks.rados.rados.0.smithi105.stdout:1411: expect (ObjNum 219 snap 21 seq_num 219) 2024-04-20T02:52:07.159 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:07.179 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:07.187 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:07.187+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:07.200 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:07.219 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:07.239 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:07.240 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:07.241 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:07.241 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:07.241 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:07.241 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:07.241 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:07.241 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:07.241 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:07.249 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:07.249 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:07.259 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:07.293 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:07.293+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:07.388 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:07.389+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:07.399 INFO:tasks.rados.rados.0.smithi105.stdout:1414: expect (ObjNum 258 snap 22 seq_num 258) 2024-04-20T02:52:07.428 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:07.428 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:07.439 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:07.489 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:07.489+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:07.517 INFO:tasks.rados.rados.0.smithi105.stdout:1413: finishing write tid 1 to smithi10548823-37 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1413: finishing write tid 2 to smithi10548823-37 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1413: finishing write tid 3 to smithi10548823-37 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1413: finishing write tid 4 to smithi10548823-37 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1413: oid 37 updating version 0 to 57 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1413: oid 37 version 57 is already newer than 55 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1413: oid 37 version 57 is already newer than 56 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 57 (ObjNum 263 snap 24 seq_num 263) dirty exists 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1413: left oid 37 (ObjNum 263 snap 24 seq_num 263) 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1411: done (4 left) 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1413: done (3 left) 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1414: done (2 left) 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1415: done (1 left) 2024-04-20T02:52:07.518 INFO:tasks.rados.rados.0.smithi105.stdout:1416: snap_create 2024-04-20T02:52:07.590 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:07.591+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:07.608 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:07.608 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:07.619 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:07.692 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:07.691+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:07.788 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:07.789 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:07.792 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:07.792+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:07.799 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:07.893 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:07.892+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:07.927 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:07.927 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:07.938 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:07.993 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:07.994+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:08.072 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:08.072 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:08.083 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:08.124 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:08.124+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:08.130 INFO:tasks.rados.rados.0.smithi105.stdout:1416: done (1 left) 2024-04-20T02:52:08.130 INFO:tasks.rados.rados.0.smithi105.stdout:1417: tier_promote oid 24 2024-04-20T02:52:08.130 INFO:tasks.rados.rados.0.smithi105.stdout:1418: rollback oid 32 current snap is 25 2024-04-20T02:52:08.130 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 32 to 23 2024-04-20T02:52:08.130 INFO:tasks.rados.rados.0.smithi105.stdout:1419: snap_remove snap 24 2024-04-20T02:52:08.196 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:08.196+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:08.253 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:08.253 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:08.263 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:08.299 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:52:08.299 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:52:08.300 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-20T02:52:08.300 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-04-20T02:52:08.300 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0 2024-04-20T02:52:08.305 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:08.305+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:08.398 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:08.399+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:08.441 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:08.441 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:08.450 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:08.499 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:08.500+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:08.581 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:08.581 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:08.591 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:08.601 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:08.601+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:08.701 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:08.702+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:08.758 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:08.758 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:08.768 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:08.803 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:08.803+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:08.897 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:08.897 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:08.903 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:08.904+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:08.907 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:09.005 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:09.005+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:09.036 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:09.036 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:09.046 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:09.106 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:09.107+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:09.135 INFO:tasks.rados.rados.0.smithi105.stdout:1419: done (3 left) 2024-04-20T02:52:09.136 INFO:tasks.rados.rados.0.smithi105.stdout:1420: tier_evict oid 43 2024-04-20T02:52:09.136 INFO:tasks.rados.rados.0.smithi105.stdout:1420: tier_evict oid 43 snap -1 2024-04-20T02:52:09.136 INFO:tasks.rados.rados.0.smithi105.stdout:1421: write (excl) oid 28 current snap is 25 2024-04-20T02:52:09.136 INFO:tasks.rados.rados.0.smithi105.stdout:1421: seq_num 265 ranges {453855=696987,1890787=548083} 2024-04-20T02:52:09.136 INFO:tasks.rados.rados.0.smithi105.stdout:1421: writing smithi10548823-28 from 453855 to 1150842 tid 1 2024-04-20T02:52:09.138 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.6 primary-affinity to 0 (802) 2024-04-20T02:52:09.144 INFO:tasks.rados.rados.0.smithi105.stdout:1421: writing smithi10548823-28 from 1890787 to 2438870 tid 2 2024-04-20T02:52:09.144 INFO:tasks.rados.rados.0.smithi105.stdout:1422: rollback oid 49 current snap is 25 2024-04-20T02:52:09.144 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 49 to 22 2024-04-20T02:52:09.144 INFO:tasks.rados.rados.0.smithi105.stdout:1423: write (excl) oid 47 current snap is 25 2024-04-20T02:52:09.144 INFO:tasks.rados.rados.0.smithi105.stdout:1423: seq_num 266 ranges {470662=772236,1832916=408094} 2024-04-20T02:52:09.151 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:52:09.155 INFO:tasks.rados.rados.0.smithi105.stdout:1423: writing smithi10548823-47 from 470662 to 1242898 tid 1 2024-04-20T02:52:09.157 INFO:tasks.rados.rados.0.smithi105.stdout:1423: writing smithi10548823-47 from 1832916 to 2241010 tid 2 2024-04-20T02:52:09.157 INFO:tasks.rados.rados.0.smithi105.stdout:1424: set_chunk oid 8 target oid 2024-04-20T02:52:09.190 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:09.190 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:09.203 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:09.211 INFO:tasks.rados.rados.0.smithi105.stdout:1424: set_chunk oid 8 offset: 108032 length: 354304 target oid d08f7bc688adc0271ae0d2edebb260dc23beac2a3f5012433cf611eb130c4e6e offset: 0 snap -1 2024-04-20T02:52:09.211 INFO:tasks.rados.rados.0.smithi105.stdout:1425: read oid 44 snap -1 2024-04-20T02:52:09.212 INFO:tasks.rados.rados.0.smithi105.stdout:1425: expect (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:52:09.212 INFO:tasks.rados.rados.0.smithi105.stdout:1426: write (excl) oid 31 current snap is 25 2024-04-20T02:52:09.212 INFO:tasks.rados.rados.0.smithi105.stdout:1418: finishing rollback tid 1 to smithi10548823-32 2024-04-20T02:52:09.212 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 129 snap 0 seq_num 61336744) dirty dne 2024-04-20T02:52:09.212 INFO:tasks.rados.rados.0.smithi105.stdout:1426: seq_num 267 ranges {487469=563838,1607751=551752} 2024-04-20T02:52:09.212 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:09.211+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:09.216 INFO:tasks.rados.rados.0.smithi105.stdout:1426: writing smithi10548823-31 from 487469 to 1051307 tid 1 2024-04-20T02:52:09.223 INFO:tasks.rados.rados.0.smithi105.stdout:1426: writing smithi10548823-31 from 1607751 to 2159503 tid 2 2024-04-20T02:52:09.223 INFO:tasks.rados.rados.0.smithi105.stdout:1418: done (9 left) 2024-04-20T02:52:09.223 INFO:tasks.rados.rados.0.smithi105.stdout:1427: read oid 21 snap -1 2024-04-20T02:52:09.223 INFO:tasks.rados.rados.0.smithi105.stdout:1427: expect deleted 2024-04-20T02:52:09.223 INFO:tasks.rados.rados.0.smithi105.stdout:1428: read oid 11 snap -1 2024-04-20T02:52:09.223 INFO:tasks.rados.rados.0.smithi105.stdout:1428: expect (ObjNum 210 snap 19 seq_num 210) 2024-04-20T02:52:09.223 INFO:tasks.rados.rados.0.smithi105.stdout:1429: set_chunk oid 42 target oid 2024-04-20T02:52:09.253 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:09.307 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:09.308+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:09.370 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:09.371 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:09.383 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:09.404 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:09.408 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:09.408+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:09.425 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:09.444 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:09.465 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:09.484 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:09.504 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:09.509 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:09.510+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:09.515 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:09.515 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:09.515 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:09.515 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:09.524 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:09.527 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:09.544 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:09.545 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:09.611 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:09.611+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:09.672 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:09.672 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:09.672 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:09.672 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:09.672 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:09.683 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:09.724 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:09.724+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:09.812 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:09.812+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:09.853 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:09.853 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:09.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:09.853 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 83.308734224999995, 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.854 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:52:09.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:52:09.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:52:09.860 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.860 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.860 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 83.302513150999999, 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:52:09.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.862 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.862 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:09.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.862 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:09.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.863 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:52:09.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:52:09.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:52:09.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:52:09.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:52:09.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:52:09.869 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.869 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.869 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 83.294112947000002, 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:09.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:52:09.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 83.293770301999999, 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 75.487448298999993, 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.886 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 75.480185336999995, 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:09.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:52:09.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:52:09.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:52:09.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:52:09.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 75.474242808, 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:52:09.892 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 75.474039816000001, 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.898 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:52:09.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:52:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 75.474016888999998, 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:09.904 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:52:09.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:52:09.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 75.463933155000007, 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:09.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:52:09.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 75.447047746999999, 2024-04-20T02:52:09.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:09.917 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 75.430771305999997, 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:09.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:52:09.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:52:09.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.921 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.406215834000001, 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:52:09.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:52:09.927 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 45.091900058, 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:52:09.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:52:09.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:52:09.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:09.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2545 4.9 e91/81 4:95e19a88:::smithi10548823-40:head v 91'253, mlcod=91'251)", 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.437262943, 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054757404000000003, 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415210+0000", 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416004+0000", 2024-04-20T02:52:09.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079381199999999997 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416022+0000", 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7969e-05 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416031+0000", 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9279999999999999e-06 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416100+0000", 2024-04-20T02:52:09.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8891000000000002e-05 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416128+0000", 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8203000000000001e-05 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469912+0000", 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053783794000000003 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469968+0000", 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6474000000000002e-05 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.504234136000001, 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.938 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:52:09.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2700 4.0 4:007b2a87:::smithi10548823-39:head [write 1967200~714747] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.495743942000001, 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061846287, 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.944 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2700 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356728+0000", 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356879+0000", 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000150958 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356894+0000", 2024-04-20T02:52:09.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5132e-05 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356906+0000", 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2071e-05 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366666+0000", 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0097595289999999994 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366774+0000", 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108527 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366956+0000", 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181359 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390515+0000", 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023559902000000001 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406790+0000", 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016274555999999999 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418488+0000", 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011697591 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418529+0000", 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1545000000000003e-05 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418576+0000", 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6903000000000001e-05 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2701 4.0 4:007b2a87:::smithi10548823-39:head [write 3421983~426820] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.48856795, 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054799176999999998, 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2701 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363904+0000", 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364154+0000", 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000250179 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364171+0000", 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6748e-05 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364182+0000", 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1371e-05 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390552+0000", 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026369607 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390652+0000", 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010059499999999999 2024-04-20T02:52:09.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390857+0000", 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020445199999999999 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418224+0000", 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027367316999999999 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418282+0000", 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8279000000000002e-05 2024-04-20T02:52:09.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418618+0000", 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000335967 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418661+0000", 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2614999999999998e-05 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418705+0000", 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3993999999999999e-05 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.955 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2702 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (57) in=7b,truncate 3848803] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.488281896, 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058669165000000002, 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2702 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364191+0000", 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364196+0000", 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4490000000000001e-06 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364201+0000", 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5040000000000002e-06 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364206+0000", 2024-04-20T02:52:09.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.405895+0000", 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041689047 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406021+0000", 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000125561 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406541+0000", 2024-04-20T02:52:09.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052000699999999996 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418451+0000", 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011910041999999999 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418747+0000", 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029622399999999999 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422704+0000", 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039571859999999997 2024-04-20T02:52:09.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422781+0000", 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.661e-05 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422861+0000", 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9987999999999994e-05 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.961 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2703 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.488262844000001, 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058940278999999998, 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2703 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:09.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:09.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364213+0000", 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4830000000000002e-06 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364220+0000", 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.007e-06 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364225+0000", 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0350000000000001e-06 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406820+0000", 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042595135999999999 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406923+0000", 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102846 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422897+0000", 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015974202999999999 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:09.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423015+0000", 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117581 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423151+0000", 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013635899999999999 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:09.966 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:09.968 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:09.968 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:09.968+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:10.069 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:10.069+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:10.110 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:10.110 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:10.110 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:10.121 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:10.133 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:52:10.151 INFO:tasks.rados.rados.0.smithi105.stdout:1429: set_chunk oid 42 offset: 908288 length: 728064 target oid 84415da68bff290300d081b382137aedb782d40022587449f40428d452b1c95b offset: 0 snap -1 2024-04-20T02:52:10.151 INFO:tasks.rados.rados.0.smithi105.stdout:1430: tier_promote oid 1 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1420: got (22) Invalid argument 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1422: finishing rollback tid 0 to smithi10548823-49 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1422: finishing rollback tid 1 to smithi10548823-49 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1422: finishing rollback tid 2 to smithi10548823-49 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 173 (ObjNum 260 snap 22 seq_num 260) dirty exists 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1423: finishing write tid 1 to smithi10548823-47 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1423: finishing write tid 2 to smithi10548823-47 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1420: done (12 left) 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1422: done (11 left) 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1431: read oid 14 snap 23 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1431: expect (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1432: read oid 23 snap 22 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1432: expect (ObjNum 224 snap 22 seq_num 224) 2024-04-20T02:52:10.152 INFO:tasks.rados.rados.0.smithi105.stdout:1433: write oid 6 current snap is 25 2024-04-20T02:52:10.153 INFO:tasks.rados.rados.0.smithi105.stdout:1433: seq_num 268 ranges {504276=639087,1666233=695410} 2024-04-20T02:52:10.157 INFO:tasks.rados.rados.0.smithi105.stdout:1433: writing smithi10548823-6 from 504276 to 1143363 tid 1 2024-04-20T02:52:10.167 INFO:tasks.rados.rados.0.smithi105.stdout:1433: writing smithi10548823-6 from 1666233 to 2361643 tid 2 2024-04-20T02:52:10.167 INFO:tasks.rados.rados.0.smithi105.stdout:1434: read oid 43 snap -1 2024-04-20T02:52:10.167 INFO:tasks.rados.rados.0.smithi105.stdout:1434: expect (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:52:10.167 INFO:tasks.rados.rados.0.smithi105.stdout:1435: read oid 4 snap -1 2024-04-20T02:52:10.167 INFO:tasks.rados.rados.0.smithi105.stdout:1435: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:10.167 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1423: finishing write tid 3 to smithi10548823-47 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1423: finishing write tid 4 to smithi10548823-47 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1423: oid 47 updating version 0 to 98 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1423: oid 47 version 98 is already newer than 97 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1423: oid 47 version 98 is already newer than 96 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 98 (ObjNum 266 snap 25 seq_num 266) dirty exists 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1423: left oid 47 (ObjNum 266 snap 25 seq_num 266) 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1421: finishing write tid 1 to smithi10548823-28 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1412: got (0) Success 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 282 (ObjNum 250 snap 22 seq_num 250) dirty exists 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1421: finishing write tid 2 to smithi10548823-28 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1421: finishing write tid 3 to smithi10548823-28 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1412: done (15 left) 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1423: done (14 left) 2024-04-20T02:52:10.168 INFO:tasks.rados.rados.0.smithi105.stdout:1436: write oid 29 current snap is 25 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:1421: finishing write tid 4 to smithi10548823-28 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:1421: oid 28 updating version 0 to 178 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:1421: oid 28 version 178 is already newer than 177 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:1421: oid 28 updating version 178 to 179 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 179 (ObjNum 265 snap 25 seq_num 265) dirty exists 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:1421: left oid 28 (ObjNum 265 snap 25 seq_num 265) 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:1417: got (0) Success 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 273 (ObjNum 234 snap 22 seq_num 234) dirty exists 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:1424: finishing set_chunk to oid 8 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 108032 length: 354304 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 0 (ObjNum 257 snap 22 seq_num 257) dirty exists 2024-04-20T02:52:10.169 INFO:tasks.rados.rados.0.smithi105.stdout:1425: expect (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:52:10.204 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:10.205+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:10.271 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:10.271+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:10.291 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:10.292 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:10.292 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:10.292 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:10.292 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:10.302 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:10.383 INFO:tasks.rados.rados.0.smithi105.stdout:1428: expect (ObjNum 210 snap 19 seq_num 210) 2024-04-20T02:52:10.385 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:10.386+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:10.472 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.528656427000001, 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.473 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:10.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.478 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.521178109000004, 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.479 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:10.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:52:10.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:52:10.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.516420210000007, 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:10.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:52:10.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:52:10.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.092991276000006, 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:10.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:52:10.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:52:10.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.496 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.092954392999999, 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:10.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:52:10.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.503 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.092929760000004, 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:10.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:52:10.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:52:10.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.508 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.086436348999996, 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:10.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.080372327000006, 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.515 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:10.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.070310562000003, 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:10.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.526 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.069617957999995, 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.527 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:10.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.532 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.069606304000004, 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:10.537 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.363541748000003, 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:52:10.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.541 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.344481032000004, 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:52:10.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:52:10.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:52:10.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.547 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.338534718000005, 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.553 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.337969276999999, 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.559 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.337943245999995, 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:52:10.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:52:10.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.564 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.335730444999999, 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:52:10.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:52:10.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.330380921, 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:52:10.569 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.573 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.329910964, 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:52:10.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:10.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.578 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.329888428, 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:10.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:52:10.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:52:10.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:10.582 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.583 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.583 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.583 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:10.583 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:10.583 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:10.585 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:1426: finishing write tid 1 to smithi10548823-31 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:1426: finishing write tid 2 to smithi10548823-31 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:1426: finishing write tid 3 to smithi10548823-31 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:1426: finishing write tid 4 to smithi10548823-31 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:1426: oid 31 updating version 0 to 175 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:1426: oid 31 updating version 175 to 176 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:1426: oid 31 updating version 176 to 177 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 177 (ObjNum 267 snap 25 seq_num 267) dirty exists 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:1426: left oid 31 (ObjNum 267 snap 25 seq_num 267) 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:1429: finishing set_chunk to oid 42 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk returned error code -95 offset: 908288 length: 728064 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 222 snap 21 seq_num 222) dirty exists 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:1430: got (0) Success 2024-04-20T02:52:10.586 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 235 (ObjNum 249 snap 22 seq_num 249) dirty exists 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1436: seq_num 269 ranges {521083=714336,1724715=555421} 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1436: writing smithi10548823-29 from 521083 to 1235419 tid 1 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1436: writing smithi10548823-29 from 1724715 to 2280136 tid 2 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1417: done (14 left) 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1421: done (13 left) 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1424: done (12 left) 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1425: done (11 left) 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1426: done (10 left) 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1427: done (9 left) 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1428: done (8 left) 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1429: done (7 left) 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1430: done (6 left) 2024-04-20T02:52:10.587 INFO:tasks.rados.rados.0.smithi105.stdout:1437: snap_create 2024-04-20T02:52:10.588 INFO:tasks.rados.rados.0.smithi105.stdout:1431: expect (ObjNum 99 snap 5 seq_num 99) 2024-04-20T02:52:10.588 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:10.584+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:10.687 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:10.687+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:10.721 INFO:tasks.rados.rados.0.smithi105.stdout:1432: expect (ObjNum 224 snap 22 seq_num 224) 2024-04-20T02:52:10.754 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:10.754 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:10.754 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:10.754 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:10.764 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:10.808 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:10.809+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:10.890 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:10.890+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:10.892 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:10.892 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:10.892 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:10.892 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:10.892 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:10.903 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:10.923 INFO:tasks.rados.rados.0.smithi105.stdout:1435: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:10.991 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:10.991+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.335607120000006, 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.072 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:52:11.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:52:11.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.077 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.304318058999996, 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.078 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:11.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:11.083 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.299627603000005, 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:52:11.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.088 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.292871133999995, 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:52:11.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:52:11.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:52:11.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:52:11.095 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.283055622999996, 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:52:11.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:52:11.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:52:11.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:52:11.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.282011690999994, 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.281990531000005, 2024-04-20T02:52:11.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:11.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.074419950000006, 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.060225305000003, 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.116 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:52:11.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:52:11.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:52:11.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:11.120 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.054258047999994, 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:52:11.124 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.045852045000004, 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:52:11.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.942220004000006, 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.132 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.711623693999996, 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:52:11.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.684679544999995, 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.659321277999993, 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.631180834000006, 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:52:11.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:52:11.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.593812228999994, 2024-04-20T02:52:11.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.153 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.593515306, 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.157 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.588432187999999, 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:52:11.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:52:11.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:52:11.161 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.561429629000003, 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:52:11.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:52:11.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:52:11.165 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.166 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.166 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.166 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.166 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.166 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:11.167 INFO:tasks.rados.rados.0.smithi105.stdout:1434: expect (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:52:11.168 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:11.168 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:11.167+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:11.268 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:11.268+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:11.297 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:11.297 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:11.297 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:11.297 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:11.306 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:11.369 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:11.370+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:11.435 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:11.435 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:11.435 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:11.435 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:11.436 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:11.445 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1433: finishing write tid 1 to smithi10548823-6 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1433: finishing write tid 2 to smithi10548823-6 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1433: finishing write tid 3 to smithi10548823-6 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1433: finishing write tid 4 to smithi10548823-6 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1433: oid 6 updating version 0 to 160 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1433: oid 6 updating version 160 to 161 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1433: oid 6 version 161 is already newer than 159 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 161 (ObjNum 268 snap 25 seq_num 268) dirty exists 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1433: left oid 6 (ObjNum 268 snap 25 seq_num 268) 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1436: finishing write tid 1 to smithi10548823-29 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1431: done (6 left) 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1432: done (5 left) 2024-04-20T02:52:11.462 INFO:tasks.rados.rados.0.smithi105.stdout:1433: done (4 left) 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1434: done (3 left) 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1435: done (2 left) 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1437: done (1 left) 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1438: read oid 5 snap -1 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1438: expect (ObjNum 262 snap 23 seq_num 262) 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1439: read oid 17 snap -1 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1439: expect deleted 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1440: read oid 16 snap 25 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1440: expect deleted 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1441: read oid 9 snap -1 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1441: expect deleted 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1436: finishing write tid 2 to smithi10548823-29 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1436: finishing write tid 3 to smithi10548823-29 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1436: finishing write tid 4 to smithi10548823-29 2024-04-20T02:52:11.463 INFO:tasks.rados.rados.0.smithi105.stdout:1436: oid 29 updating version 0 to 29 2024-04-20T02:52:11.464 INFO:tasks.rados.rados.0.smithi105.stdout:1436: oid 29 updating version 29 to 31 2024-04-20T02:52:11.464 INFO:tasks.rados.rados.0.smithi105.stdout:1436: oid 29 version 31 is already newer than 30 2024-04-20T02:52:11.464 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 31 (ObjNum 269 snap 25 seq_num 269) dirty exists 2024-04-20T02:52:11.464 INFO:tasks.rados.rados.0.smithi105.stdout:1436: left oid 29 (ObjNum 269 snap 25 seq_num 269) 2024-04-20T02:52:11.464 INFO:tasks.rados.rados.0.smithi105.stdout:1442: read oid 43 snap 25 2024-04-20T02:52:11.464 INFO:tasks.rados.rados.0.smithi105.stdout:1442: expect (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:52:11.464 INFO:tasks.rados.rados.0.smithi105.stdout:1436: done (5 left) 2024-04-20T02:52:11.464 INFO:tasks.rados.rados.0.smithi105.stdout:1443: write oid 42 current snap is 26 2024-04-20T02:52:11.464 INFO:tasks.rados.rados.0.smithi105.stdout:1443: seq_num 270 ranges {537890=789585,1783197=699079} 2024-04-20T02:52:11.468 INFO:tasks.rados.rados.0.smithi105.stdout:1443: writing smithi10548823-42 from 537890 to 1327475 tid 1 2024-04-20T02:52:11.475 INFO:tasks.rados.rados.0.smithi105.stdout:1443: writing smithi10548823-42 from 1783197 to 2482276 tid 2 2024-04-20T02:52:11.475 INFO:tasks.rados.rados.0.smithi105.stdout:1444: read oid 13 snap -1 2024-04-20T02:52:11.475 INFO:tasks.rados.rados.0.smithi105.stdout:1444: expect (ObjNum 151 snap 10 seq_num 151) 2024-04-20T02:52:11.475 INFO:tasks.rados.rados.0.smithi105.stdout:1445: write oid 11 current snap is 26 2024-04-20T02:52:11.475 INFO:tasks.rados.rados.0.smithi105.stdout:1442: expect (ObjNum 213 snap 20 seq_num 213) 2024-04-20T02:52:11.486 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:11.487+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:11.557 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:11.571 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:11.571+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:11.618 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:11.618 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:11.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:11.618 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:11.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.618 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.013403369, 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:11.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:52:11.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:52:11.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.249536969999994, 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:52:11.629 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.136233928999999, 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:52:11.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:52:11.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.130468411999999, 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:52:11.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:52:11.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.776448277, 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.766084672000005, 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.645 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:11.649 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.758000193000001, 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:52:11.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:52:11.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.476441968000003, 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:11.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:52:11.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:52:11.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:52:11.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.471304857999996, 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:11.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:52:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.471031846000002, 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.471002460999998, 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:52:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:52:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.162108496000002, 2024-04-20T02:52:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035060916999999997, 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456603+0000", 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001394295 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456616+0000", 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2605999999999999e-05 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456631+0000", 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4735e-05 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456682+0000", 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1048000000000003e-05 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456705+0000", 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2881e-05 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490235+0000", 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033530250999999997 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490270+0000", 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5379000000000001e-05 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:52:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 48.783675162000002, 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:52:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:52:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2120 4.11 e70/65 4:8b91c759:::smithi10548823-5:head v 70'113, mlcod=67'111)", 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:27.728792+0000", 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 43.888526077999998, 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033453240000000002, 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.728792+0000", 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.728792+0000", 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.728791+0000", 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.730196+0000", 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014048140000000001 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.730208+0000", 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1637999999999999e-05 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.730216+0000", 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5159999999999994e-06 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.730269+0000", 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2525999999999998e-05 2024-04-20T02:52:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.730293+0000", 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3929000000000001e-05 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.762205+0000", 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031912325999999998 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.762245+0000", 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9826999999999997e-05 2024-04-20T02:52:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2132 4.10 e70/63 4:0b0a7da5:::smithi10548823-43:head v 70'76, mlcod=65'73)", 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 43.873077123000002, 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036823724000000002, 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744239+0000", 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744631+0000", 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039175199999999999 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744644+0000", 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3079e-05 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744653+0000", 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8629999999999997e-06 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744706+0000", 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2627999999999998e-05 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744737+0000", 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1387999999999998e-05 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781039+0000", 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036302147 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781064+0000", 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5182999999999999e-05 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2293 4.3 e76/60 4:c07c6daa:::smithi10548823-7:head v 76'165, mlcod=64'161)", 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.428209099999997, 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034106808000000002, 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189108+0000", 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190758+0000", 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016504740000000001 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190778+0000", 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9507000000000001e-05 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190787+0000", 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1740000000000006e-06 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190836+0000", 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9351999999999998e-05 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190854+0000", 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7629e-05 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.223165+0000", 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032310885999999997 2024-04-20T02:52:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.223216+0000", 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0615000000000002e-05 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.336194939999999, 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:52:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:52:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:52:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.318926506, 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:52:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.14 98/97 [PushOp(4:2e88ea27:::smithi10548823-12:head, version: 98'33, data_included: [406769~907695,1439434~140881], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:2e88ea27:::smithi10548823-12:head@98'33, size: 2033680, copy_subset: [0~2033680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1580315, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 15.367159672, 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033699081999999998, 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251322+0000", 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001164251 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251340+0000", 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.789e-05 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251356+0000", 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5962000000000001e-05 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251428+0000", 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2750000000000007e-05 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251444+0000", 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5248e-05 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.283857+0000", 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032413659999999997 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.6.0:82 4.15 e104/103 4:a9897a5e:::smithi10548823-42:head v 104'64, mlcod=93'62)", 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.4814453780000001, 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035573920000000002, 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135871+0000", 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136395+0000", 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00052370100000000001 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136409+0000", 2024-04-20T02:52:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4191e-05 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136418+0000", 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0100000000000001e-06 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136468+0000", 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9458e-05 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136499+0000", 2024-04-20T02:52:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1062999999999997e-05 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.171417+0000", 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034918607999999997 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.171446+0000", 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9000999999999999e-05 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:11.716 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:11.717 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:11.717 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:11.718 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:11.718+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:11.722 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:11.740 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:11.759 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:11.777 INFO:tasks.rados.rados.0.smithi105.stdout:1438: expect (ObjNum 262 snap 23 seq_num 262) 2024-04-20T02:52:11.779 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:11.798 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:11.817 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:11.818 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:11.818+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:11.837 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:11.838 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:11.838 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:11.838 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:11.838 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:11.838 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:11.838 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:11.838 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:11.838 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:11.888 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:11.888 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:11.899 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:11.905 INFO:tasks.rados.rados.0.smithi105.stdout:1444: expect (ObjNum 151 snap 10 seq_num 151) 2024-04-20T02:52:11.918 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:11.918+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:12.019 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:12.019+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:12.026 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:12.026 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:12.037 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:12.060 INFO:tasks.rados.rados.0.smithi105.stdout:1443: finishing write tid 1 to smithi10548823-42 2024-04-20T02:52:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:1443: finishing write tid 2 to smithi10548823-42 2024-04-20T02:52:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:1443: finishing write tid 3 to smithi10548823-42 2024-04-20T02:52:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:1443: finishing write tid 4 to smithi10548823-42 2024-04-20T02:52:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:1443: oid 42 updating version 0 to 69 2024-04-20T02:52:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:1443: oid 42 updating version 69 to 70 2024-04-20T02:52:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:1443: oid 42 version 70 is already newer than 68 2024-04-20T02:52:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 70 (ObjNum 270 snap 26 seq_num 270) dirty exists 2024-04-20T02:52:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:1443: left oid 42 (ObjNum 270 snap 26 seq_num 270) 2024-04-20T02:52:12.061 INFO:tasks.rados.rados.0.smithi105.stdout:1445: seq_num 271 ranges {554697=464834,1725326=559090} 2024-04-20T02:52:12.065 INFO:tasks.rados.rados.0.smithi105.stdout:1445: writing smithi10548823-11 from 554697 to 1019531 tid 1 2024-04-20T02:52:12.073 INFO:tasks.rados.rados.0.smithi105.stdout:1445: writing smithi10548823-11 from 1725326 to 2284416 tid 2 2024-04-20T02:52:12.073 INFO:tasks.rados.rados.0.smithi105.stdout:1438: done (7 left) 2024-04-20T02:52:12.073 INFO:tasks.rados.rados.0.smithi105.stdout:1440: done (6 left) 2024-04-20T02:52:12.073 INFO:tasks.rados.rados.0.smithi105.stdout:1441: done (5 left) 2024-04-20T02:52:12.073 INFO:tasks.rados.rados.0.smithi105.stdout:1442: done (4 left) 2024-04-20T02:52:12.073 INFO:tasks.rados.rados.0.smithi105.stdout:1443: done (3 left) 2024-04-20T02:52:12.074 INFO:tasks.rados.rados.0.smithi105.stdout:1444: done (2 left) 2024-04-20T02:52:12.074 INFO:tasks.rados.rados.0.smithi105.stdout:1446: write oid 2 current snap is 26 2024-04-20T02:52:12.074 INFO:tasks.rados.rados.0.smithi105.stdout:1446: seq_num 272 ranges {571504=540083,1783808=702748} 2024-04-20T02:52:12.079 INFO:tasks.rados.rados.0.smithi105.stdout:1446: writing smithi10548823-2 from 571504 to 1111587 tid 1 2024-04-20T02:52:12.087 INFO:tasks.rados.rados.0.smithi105.stdout:1446: writing smithi10548823-2 from 1783808 to 2486556 tid 2 2024-04-20T02:52:12.087 INFO:tasks.rados.rados.0.smithi105.stdout:1447: read oid 46 snap 22 2024-04-20T02:52:12.087 INFO:tasks.rados.rados.0.smithi105.stdout:1447: expect (ObjNum 232 snap 22 seq_num 232) 2024-04-20T02:52:12.088 INFO:tasks.rados.rados.0.smithi105.stdout:1448: write (excl) oid 22 current snap is 26 2024-04-20T02:52:12.088 INFO:tasks.rados.rados.0.smithi105.stdout:1448: seq_num 273 ranges {588311=615332,1842290=562759} 2024-04-20T02:52:12.095 INFO:tasks.rados.rados.0.smithi105.stdout:1448: writing smithi10548823-22 from 588311 to 1203643 tid 1 2024-04-20T02:52:12.102 INFO:tasks.rados.rados.0.smithi105.stdout:1448: writing smithi10548823-22 from 1842290 to 2405049 tid 2 2024-04-20T02:52:12.102 INFO:tasks.rados.rados.0.smithi105.stdout:1449: write oid 14 current snap is 26 2024-04-20T02:52:12.102 INFO:tasks.rados.rados.0.smithi105.stdout:1449: seq_num 274 ranges {605118=406934,1617125=706417} 2024-04-20T02:52:12.108 INFO:tasks.rados.rados.0.smithi105.stdout:1449: writing smithi10548823-14 from 605118 to 1012052 tid 1 2024-04-20T02:52:12.117 INFO:tasks.rados.rados.0.smithi105.stdout:1449: writing smithi10548823-14 from 1617125 to 2323542 tid 2 2024-04-20T02:52:12.117 INFO:tasks.rados.rados.0.smithi105.stdout:1450: read oid 16 snap 22 2024-04-20T02:52:12.117 INFO:tasks.rados.rados.0.smithi105.stdout:1450: expect deleted 2024-04-20T02:52:12.117 INFO:tasks.rados.rados.0.smithi105.stdout:1451: read oid 50 snap -1 2024-04-20T02:52:12.117 INFO:tasks.rados.rados.0.smithi105.stdout:1451: expect (ObjNum 221 snap 21 seq_num 221) 2024-04-20T02:52:12.118 INFO:tasks.rados.rados.0.smithi105.stdout:1452: read oid 45 snap -1 2024-04-20T02:52:12.118 INFO:tasks.rados.rados.0.smithi105.stdout:1452: expect (ObjNum 220 snap 21 seq_num 220) 2024-04-20T02:52:12.118 INFO:tasks.rados.rados.0.smithi105.stdout:1453: set_chunk oid 44 target oid 2024-04-20T02:52:12.120 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:12.120+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:12.146 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:52:12.154 INFO:tasks.rados.rados.0.smithi105.stdout:1453: set_chunk oid 44 offset: 1078784 length: 295424 target oid 9fb82b4dd01a97fd16845ff81d25092fbab5c875f8d777346657494cc1c223f6 offset: 0 snap -1 2024-04-20T02:52:12.154 INFO:tasks.rados.rados.0.smithi105.stdout:1454: write oid 39 current snap is 26 2024-04-20T02:52:12.154 INFO:tasks.rados.rados.0.smithi105.stdout:1454: seq_num 275 ranges {621925=482183,1559254=566428} 2024-04-20T02:52:12.160 INFO:tasks.rados.rados.0.smithi105.stdout:1454: writing smithi10548823-39 from 621925 to 1104108 tid 1 2024-04-20T02:52:12.166 INFO:tasks.rados.rados.0.smithi105.stdout:1454: writing smithi10548823-39 from 1559254 to 2125682 tid 2 2024-04-20T02:52:12.167 INFO:tasks.rados.rados.0.smithi105.stdout:1455: set_chunk oid 41 target oid 2024-04-20T02:52:12.167 INFO:tasks.rados.rados.0.smithi105.stdout:1455: set_chunk oid 41 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:12.167 INFO:tasks.rados.rados.0.smithi105.stdout:1456: set_chunk oid 13 target oid 2024-04-20T02:52:12.178 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:12.178 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:12.190 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:12.223 INFO:tasks.rados.rados.0.smithi105.stdout:1456: set_chunk oid 13 offset: 221696 length: 2167296 target oid 9959c197750272dcc80f7792c7e9c2ad984a61782def66786ba6544c17994c48 offset: 0 snap -1 2024-04-20T02:52:12.223 INFO:tasks.rados.rados.0.smithi105.stdout:1457: tier_promote oid 5 2024-04-20T02:52:12.223 INFO:tasks.rados.rados.0.smithi105.stdout:1447: expect (ObjNum 232 snap 22 seq_num 232) 2024-04-20T02:52:12.231 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:12.232+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:12.322 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:12.323+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:12.331 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:12.332 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:12.342 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:12.422 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:12.423+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:12.445 INFO:tasks.rados.rados.0.smithi105.stdout:1445: finishing write tid 1 to smithi10548823-11 2024-04-20T02:52:12.445 INFO:tasks.rados.rados.0.smithi105.stdout:1445: finishing write tid 2 to smithi10548823-11 2024-04-20T02:52:12.445 INFO:tasks.rados.rados.0.smithi105.stdout:1445: finishing write tid 3 to smithi10548823-11 2024-04-20T02:52:12.445 INFO:tasks.rados.rados.0.smithi105.stdout:1445: finishing write tid 4 to smithi10548823-11 2024-04-20T02:52:12.445 INFO:tasks.rados.rados.0.smithi105.stdout:1445: oid 11 updating version 0 to 115 2024-04-20T02:52:12.445 INFO:tasks.rados.rados.0.smithi105.stdout:1445: oid 11 updating version 115 to 116 2024-04-20T02:52:12.445 INFO:tasks.rados.rados.0.smithi105.stdout:1445: oid 11 updating version 116 to 117 2024-04-20T02:52:12.445 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 117 (ObjNum 271 snap 26 seq_num 271) dirty exists 2024-04-20T02:52:12.445 INFO:tasks.rados.rados.0.smithi105.stdout:1445: left oid 11 (ObjNum 271 snap 26 seq_num 271) 2024-04-20T02:52:12.445 INFO:tasks.rados.rados.0.smithi105.stdout:1445: done (13 left) 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1447: done (12 left) 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1450: done (11 left) 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1458: write oid 43 current snap is 26 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1448: finishing write tid 1 to smithi10548823-22 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1448: finishing write tid 2 to smithi10548823-22 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1448: finishing write tid 3 to smithi10548823-22 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1448: finishing write tid 4 to smithi10548823-22 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1448: oid 22 updating version 0 to 87 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1448: oid 22 updating version 87 to 89 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1448: oid 22 version 89 is already newer than 88 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 89 (ObjNum 273 snap 26 seq_num 273) dirty exists 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1448: left oid 22 (ObjNum 273 snap 26 seq_num 273) 2024-04-20T02:52:12.446 INFO:tasks.rados.rados.0.smithi105.stdout:1452: expect (ObjNum 220 snap 21 seq_num 220) 2024-04-20T02:52:12.513 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:12.513 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:12.523 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:12.524 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:12.524+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:12.622 INFO:tasks.rados.rados.0.smithi105.stdout:1449: finishing write tid 1 to smithi10548823-14 2024-04-20T02:52:12.623 INFO:tasks.rados.rados.0.smithi105.stdout:1449: finishing write tid 2 to smithi10548823-14 2024-04-20T02:52:12.623 INFO:tasks.rados.rados.0.smithi105.stdout:1449: finishing write tid 3 to smithi10548823-14 2024-04-20T02:52:12.623 INFO:tasks.rados.rados.0.smithi105.stdout:1458: seq_num 276 ranges {638732=557432,1617736=710086} 2024-04-20T02:52:12.624 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:12.624+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:12.629 INFO:tasks.rados.rados.0.smithi105.stdout:1458: writing smithi10548823-43 from 638732 to 1196164 tid 1 2024-04-20T02:52:12.638 INFO:tasks.rados.rados.0.smithi105.stdout:1458: writing smithi10548823-43 from 1617736 to 2327822 tid 2 2024-04-20T02:52:12.638 INFO:tasks.rados.rados.0.smithi105.stdout:1448: done (11 left) 2024-04-20T02:52:12.638 INFO:tasks.rados.rados.0.smithi105.stdout:1452: done (10 left) 2024-04-20T02:52:12.638 INFO:tasks.rados.rados.0.smithi105.stdout:1459: read oid 21 snap -1 2024-04-20T02:52:12.638 INFO:tasks.rados.rados.0.smithi105.stdout:1459: expect deleted 2024-04-20T02:52:12.638 INFO:tasks.rados.rados.0.smithi105.stdout:1460: delete oid 12 current snap is 26 2024-04-20T02:52:12.639 INFO:tasks.rados.rados.0.smithi105.stdout:1449: finishing write tid 4 to smithi10548823-14 2024-04-20T02:52:12.639 INFO:tasks.rados.rados.0.smithi105.stdout:1449: oid 14 updating version 0 to 107 2024-04-20T02:52:12.639 INFO:tasks.rados.rados.0.smithi105.stdout:1449: oid 14 version 107 is already newer than 105 2024-04-20T02:52:12.639 INFO:tasks.rados.rados.0.smithi105.stdout:1449: oid 14 version 107 is already newer than 106 2024-04-20T02:52:12.639 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 107 (ObjNum 274 snap 26 seq_num 274) dirty exists 2024-04-20T02:52:12.639 INFO:tasks.rados.rados.0.smithi105.stdout:1449: left oid 14 (ObjNum 274 snap 26 seq_num 274) 2024-04-20T02:52:12.639 INFO:tasks.rados.rados.0.smithi105.stdout:1451: expect (ObjNum 221 snap 21 seq_num 221) 2024-04-20T02:52:12.664 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:12.664 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:12.675 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:12.725 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:12.726+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:12.827 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:12.827+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:1446: finishing write tid 1 to smithi10548823-2 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:1446: finishing write tid 2 to smithi10548823-2 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:1446: finishing write tid 3 to smithi10548823-2 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:1446: finishing write tid 4 to smithi10548823-2 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:1446: oid 2 updating version 0 to 180 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:1446: oid 2 updating version 180 to 181 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:1446: oid 2 version 181 is already newer than 179 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 181 (ObjNum 272 snap 26 seq_num 272) dirty exists 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:1446: left oid 2 (ObjNum 272 snap 26 seq_num 272) 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:1453: finishing set_chunk to oid 44 2024-04-20T02:52:12.845 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 72 (ObjNum 208 snap 19 seq_num 208) dirty exists 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1455: finishing set_chunk to oid 41 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1455: got expected ENOENT (src dne) 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1454: finishing write tid 1 to smithi10548823-39 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1446: done (11 left) 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1449: done (10 left) 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1451: done (9 left) 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1453: done (8 left) 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1455: done (7 left) 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1460: done (6 left) 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1461: set_chunk oid 32 target oid 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1461: set_chunk oid 32 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1462: read oid 38 snap -1 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1462: expect deleted 2024-04-20T02:52:12.846 INFO:tasks.rados.rados.0.smithi105.stdout:1463: set_chunk oid 27 target oid 2024-04-20T02:52:12.847 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:12.847 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:12.857 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:12.874 INFO:tasks.rados.rados.0.smithi105.stdout:1463: set_chunk oid 27 offset: 156672 length: 1435136 target oid 744ce95c903ca5843e8a8fbe098ca5b632e32314ce6d7d87a4338f5cf423bfd1 offset: 0 snap -1 2024-04-20T02:52:12.874 INFO:tasks.rados.rados.0.smithi105.stdout:1464: tier_evict oid 1 2024-04-20T02:52:12.874 INFO:tasks.rados.rados.0.smithi105.stdout:1464: tier_evict oid 1 snap -1 2024-04-20T02:52:12.874 INFO:tasks.rados.rados.0.smithi105.stdout:1454: finishing write tid 2 to smithi10548823-39 2024-04-20T02:52:12.874 INFO:tasks.rados.rados.0.smithi105.stdout:1454: finishing write tid 3 to smithi10548823-39 2024-04-20T02:52:12.874 INFO:tasks.rados.rados.0.smithi105.stdout:1454: finishing write tid 4 to smithi10548823-39 2024-04-20T02:52:12.874 INFO:tasks.rados.rados.0.smithi105.stdout:1454: oid 39 updating version 0 to 119 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:1454: oid 39 updating version 119 to 120 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:1454: oid 39 updating version 120 to 121 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 121 (ObjNum 275 snap 26 seq_num 275) dirty exists 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:1454: left oid 39 (ObjNum 275 snap 26 seq_num 275) 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:1456: finishing set_chunk to oid 13 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 13 set_chunk returned error code -95 offset: 221696 length: 2167296 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 0 (ObjNum 151 snap 10 seq_num 151) dirty exists 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:1454: done (9 left) 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:1456: done (8 left) 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:1465: read oid 16 snap -1 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:1465: expect deleted 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:1466: write (excl) oid 19 current snap is 26 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:1457: got (0) Success 2024-04-20T02:52:12.875 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 175 (ObjNum 262 snap 23 seq_num 262) dirty exists 2024-04-20T02:52:12.876 INFO:tasks.rados.rados.0.smithi105.stdout:1458: finishing write tid 1 to smithi10548823-43 2024-04-20T02:52:12.876 INFO:tasks.rados.rados.0.smithi105.stdout:1466: seq_num 278 ranges {672346=707930,2134700=537646} 2024-04-20T02:52:12.881 INFO:tasks.rados.rados.0.smithi105.stdout:1466: writing smithi10548823-19 from 672346 to 1380276 tid 1 2024-04-20T02:52:12.889 INFO:tasks.rados.rados.0.smithi105.stdout:1466: writing smithi10548823-19 from 2134700 to 2672346 tid 2 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1457: done (9 left) 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1467: rollback oid 47 current snap is 26 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 47 to 22 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1458: finishing write tid 2 to smithi10548823-43 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1458: finishing write tid 3 to smithi10548823-43 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1459: done (9 left) 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1468: write oid 25 current snap is 26 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1458: finishing write tid 4 to smithi10548823-43 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1458: oid 43 updating version 0 to 111 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1458: oid 43 version 111 is already newer than 109 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1458: oid 43 version 111 is already newer than 110 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 111 (ObjNum 276 snap 26 seq_num 276) dirty exists 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1458: left oid 43 (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:52:12.890 INFO:tasks.rados.rados.0.smithi105.stdout:1468: seq_num 279 ranges {689153=783179,2076829=457413} 2024-04-20T02:52:12.897 INFO:tasks.rados.rados.0.smithi105.stdout:1468: writing smithi10548823-25 from 689153 to 1472332 tid 1 2024-04-20T02:52:12.904 INFO:tasks.rados.rados.0.smithi105.stdout:1468: writing smithi10548823-25 from 2076829 to 2534242 tid 2 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1458: done (9 left) 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1469: read oid 28 snap -1 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1469: expect (ObjNum 265 snap 25 seq_num 265) 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1470: read oid 29 snap 22 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1470: expect (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1471: read oid 30 snap -1 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1471: expect (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1472: read oid 44 snap -1 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1472: expect (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1473: write oid 8 current snap is 26 2024-04-20T02:52:12.905 INFO:tasks.rados.rados.0.smithi105.stdout:1473: seq_num 280 ranges {705960=458428,1735311=717424} 2024-04-20T02:52:12.912 INFO:tasks.rados.rados.0.smithi105.stdout:1473: writing smithi10548823-8 from 705960 to 1164388 tid 1 2024-04-20T02:52:12.920 INFO:tasks.rados.rados.0.smithi105.stdout:1473: writing smithi10548823-8 from 1735311 to 2452735 tid 2 2024-04-20T02:52:12.920 INFO:tasks.rados.rados.0.smithi105.stdout:1474: set_chunk oid 42 target oid 2024-04-20T02:52:12.926 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:12.927+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:1474: set_chunk oid 42 offset: 581120 length: 983040 target oid 0fd50b3e2dad8ef4668b97b461b74e5a296f4868ba0dabdb41701702a1af94f2 offset: 0 snap -1 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:1475: read oid 22 snap -1 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:1475: expect (ObjNum 273 snap 26 seq_num 273) 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:1461: finishing set_chunk to oid 32 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:1461: got expected ENOENT (src dne) 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:1463: finishing set_chunk to oid 27 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 156672 length: 1435136 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 0 (ObjNum 247 snap 22 seq_num 247) dirty exists 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:1464: got (22) Invalid argument 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:1461: done (15 left) 2024-04-20T02:52:12.961 INFO:tasks.rados.rados.0.smithi105.stdout:1462: done (14 left) 2024-04-20T02:52:12.962 INFO:tasks.rados.rados.0.smithi105.stdout:1463: done (13 left) 2024-04-20T02:52:12.962 INFO:tasks.rados.rados.0.smithi105.stdout:1464: done (12 left) 2024-04-20T02:52:12.962 INFO:tasks.rados.rados.0.smithi105.stdout:1476: set_chunk oid 35 target oid 2024-04-20T02:52:12.966 INFO:tasks.rados.rados.0.smithi105.stdout:1476: set_chunk oid 35 offset: 1038848 length: 158208 target oid a17720e16ae748cb8ff3e0fa62ef46e1a91c2a00f6efe19767d6da18990d2314 offset: 0 snap -1 2024-04-20T02:52:12.966 INFO:tasks.rados.rados.0.smithi105.stdout:1477: read oid 46 snap -1 2024-04-20T02:52:12.966 INFO:tasks.rados.rados.0.smithi105.stdout:1477: expect (ObjNum 232 snap 22 seq_num 232) 2024-04-20T02:52:12.966 INFO:tasks.rados.rados.0.smithi105.stdout:1478: write (excl) oid 6 current snap is 26 2024-04-20T02:52:12.966 INFO:tasks.rados.rados.0.smithi105.stdout:1478: seq_num 281 ranges {722767=533677,1793793=461082} 2024-04-20T02:52:12.973 INFO:tasks.rados.rados.0.smithi105.stdout:1478: writing smithi10548823-6 from 722767 to 1256444 tid 1 2024-04-20T02:52:12.979 INFO:tasks.rados.rados.0.smithi105.stdout:1478: writing smithi10548823-6 from 1793793 to 2254875 tid 2 2024-04-20T02:52:12.979 INFO:tasks.rados.rados.0.smithi105.stdout:1465: done (14 left) 2024-04-20T02:52:12.979 INFO:tasks.rados.rados.0.smithi105.stdout:1479: snap_create 2024-04-20T02:52:12.979 INFO:tasks.rados.rados.0.smithi105.stdout:1470: expect (ObjNum 201 snap 19 seq_num 201) 2024-04-20T02:52:13.027 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:13.028+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:13.029 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:13.030 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:13.041 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:13.128 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:13.128+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:13.145 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:52:13.222 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:13.222 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:13.228 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:13.229+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:13.231 INFO:tasks.rados.rados.0.smithi105.stdout:1471: expect (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:52:13.232 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:13.330 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:13.330+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:13.363 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:13.363 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:13.374 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:13.431 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:13.431+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:13.504 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:13.504 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:13.514 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:13.556 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:13.556+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:13.583 INFO:tasks.rados.rados.0.smithi105.stdout:1472: expect (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:52:13.633 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:13.634+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:13.684 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:13.684 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:13.695 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:13.734 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:13.735+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:13.804 INFO:tasks.rados.rados.0.smithi105.stdout:1466: finishing write tid 1 to smithi10548823-19 2024-04-20T02:52:13.804 INFO:tasks.rados.rados.0.smithi105.stdout:1466: finishing write tid 2 to smithi10548823-19 2024-04-20T02:52:13.804 INFO:tasks.rados.rados.0.smithi105.stdout:1466: finishing write tid 3 to smithi10548823-19 2024-04-20T02:52:13.804 INFO:tasks.rados.rados.0.smithi105.stdout:1466: finishing write tid 4 to smithi10548823-19 2024-04-20T02:52:13.804 INFO:tasks.rados.rados.0.smithi105.stdout:1466: oid 19 updating version 0 to 114 2024-04-20T02:52:13.805 INFO:tasks.rados.rados.0.smithi105.stdout:1466: oid 19 updating version 114 to 116 2024-04-20T02:52:13.805 INFO:tasks.rados.rados.0.smithi105.stdout:1466: oid 19 version 116 is already newer than 115 2024-04-20T02:52:13.805 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 116 (ObjNum 278 snap 26 seq_num 278) dirty exists 2024-04-20T02:52:13.805 INFO:tasks.rados.rados.0.smithi105.stdout:1466: left oid 19 (ObjNum 278 snap 26 seq_num 278) 2024-04-20T02:52:13.805 INFO:tasks.rados.rados.0.smithi105.stdout:1468: finishing write tid 1 to smithi10548823-25 2024-04-20T02:52:13.805 INFO:tasks.rados.rados.0.smithi105.stdout:1469: expect (ObjNum 265 snap 25 seq_num 265) 2024-04-20T02:52:13.835 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:13.835+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:13.850 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:13.863 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:13.863 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:13.876 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:13.959 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:13.960+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:13.998 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:1468: finishing write tid 2 to smithi10548823-25 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:1468: finishing write tid 3 to smithi10548823-25 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:1468: finishing write tid 4 to smithi10548823-25 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:1468: oid 25 updating version 0 to 165 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:1468: oid 25 version 165 is already newer than 164 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:1468: oid 25 version 165 is already newer than 163 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 165 (ObjNum 279 snap 26 seq_num 279) dirty exists 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:1468: left oid 25 (ObjNum 279 snap 26 seq_num 279) 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:1474: finishing set_chunk to oid 42 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 70 (ObjNum 270 snap 26 seq_num 270) dirty exists 2024-04-20T02:52:14.004 INFO:tasks.rados.rados.0.smithi105.stdout:1475: expect (ObjNum 273 snap 26 seq_num 273) 2024-04-20T02:52:14.017 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:14.037 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:14.037 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:14.038+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:14.048 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:14.048 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:14.055 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:14.059 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:14.076 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:14.098 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:14.119 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:14.138 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:14.138+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:14.139 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:14.140 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:14.140 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:14.140 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:14.140 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:14.140 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:14.140 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:14.141 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:14.141 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:14.156 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:52:14.156 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:52:14.156 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:52:14.177 INFO:tasks.rados.rados.0.smithi105.stdout:1477: expect (ObjNum 232 snap 22 seq_num 232) 2024-04-20T02:52:14.221 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:14.221 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:14.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 87.676732509000004, 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.222 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:52:14.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 87.670511434999995, 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.228 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:52:14.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:14.234 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 87.662111230999997, 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:52:14.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:52:14.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:52:14.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:52:14.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:52:14.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:52:14.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:52:14.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:52:14.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 87.661768585999994, 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:52:14.243 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.244 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:14.244 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.244 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.244 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.244 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:52:14.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.244 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:14.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:14.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:14.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:14.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:52:14.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:52:14.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:52:14.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:52:14.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:52:14.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:52:14.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:52:14.253 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.855446583000003, 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.254 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:14.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:52:14.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:52:14.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:52:14.260 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.261 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.261 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.261 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:14.261 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:14.261 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.848183621000004, 2024-04-20T02:52:14.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:52:14.262 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.262 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.262 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.262 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.262 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.262 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:52:14.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.262 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:52:14.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:52:14.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:52:14.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:52:14.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:52:14.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.842241091999995, 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.269 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:52:14.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.842038099999996, 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.275 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:52:14.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:52:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:52:14.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:52:14.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.842015172999993, 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.281 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.285 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.831931439000002, 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:52:14.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:52:14.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:52:14.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.815046030999994, 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:52:14.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.798769590000006, 2024-04-20T02:52:14.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:52:14.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:52:14.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.298 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.774214118000003, 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:14.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:52:14.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:52:14.304 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 49.459898342000002, 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:52:14.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:52:14.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:52:14.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:52:14.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2545 4.9 e91/81 4:95e19a88:::smithi10548823-40:head v 91'253, mlcod=91'251)", 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.805261226999999, 2024-04-20T02:52:14.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054757404000000003, 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415210+0000", 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416004+0000", 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079381199999999997 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416022+0000", 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7969e-05 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416031+0000", 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9279999999999999e-06 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416100+0000", 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8891000000000002e-05 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416128+0000", 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8203000000000001e-05 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469912+0000", 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053783794000000003 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469968+0000", 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6474000000000002e-05 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.314 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.87223242, 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:14.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:52:14.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2700 4.0 4:007b2a87:::smithi10548823-39:head [write 1967200~714747] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:14.320 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.863742225999999, 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061846287, 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2700 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:14.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356728+0000", 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356879+0000", 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000150958 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356894+0000", 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5132e-05 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356906+0000", 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2071e-05 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366666+0000", 2024-04-20T02:52:14.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0097595289999999994 2024-04-20T02:52:14.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366774+0000", 2024-04-20T02:52:14.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108527 2024-04-20T02:52:14.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366956+0000", 2024-04-20T02:52:14.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181359 2024-04-20T02:52:14.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390515+0000", 2024-04-20T02:52:14.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023559902000000001 2024-04-20T02:52:14.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406790+0000", 2024-04-20T02:52:14.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016274555999999999 2024-04-20T02:52:14.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418488+0000", 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011697591 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418529+0000", 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1545000000000003e-05 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418576+0000", 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6903000000000001e-05 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2701 4.0 4:007b2a87:::smithi10548823-39:head [write 3421983~426820] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.856566233999999, 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054799176999999998, 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.488 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2701 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363904+0000", 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364154+0000", 2024-04-20T02:52:14.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000250179 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364171+0000", 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6748e-05 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364182+0000", 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1371e-05 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390552+0000", 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026369607 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390652+0000", 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010059499999999999 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390857+0000", 2024-04-20T02:52:14.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020445199999999999 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418224+0000", 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027367316999999999 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418282+0000", 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8279000000000002e-05 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418618+0000", 2024-04-20T02:52:14.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000335967 2024-04-20T02:52:14.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418661+0000", 2024-04-20T02:52:14.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2614999999999998e-05 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418705+0000", 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3993999999999999e-05 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2702 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (57) in=7b,truncate 3848803] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:14.495 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.856280179999999, 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058669165000000002, 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2702 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:14.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:14.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364191+0000", 2024-04-20T02:52:14.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:14.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364196+0000", 2024-04-20T02:52:14.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4490000000000001e-06 2024-04-20T02:52:14.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364201+0000", 2024-04-20T02:52:14.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5040000000000002e-06 2024-04-20T02:52:14.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364206+0000", 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.405895+0000", 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041689047 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406021+0000", 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000125561 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406541+0000", 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052000699999999996 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418451+0000", 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011910041999999999 2024-04-20T02:52:14.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418747+0000", 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029622399999999999 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422704+0000", 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039571859999999997 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422781+0000", 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.661e-05 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422861+0000", 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9987999999999994e-05 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.504 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2703 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.856261128, 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058940278999999998, 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2703 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:14.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364213+0000", 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4830000000000002e-06 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364220+0000", 2024-04-20T02:52:14.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.007e-06 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364225+0000", 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0350000000000001e-06 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406820+0000", 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042595135999999999 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406923+0000", 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102846 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422897+0000", 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015974202999999999 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423015+0000", 2024-04-20T02:52:14.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117581 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423151+0000", 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013635899999999999 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:14.510 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:14.512 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:14.513 INFO:tasks.rados.rados.0.smithi105.stdout:1476: finishing set_chunk to oid 35 2024-04-20T02:52:14.513 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 29 (ObjNum 124 snap 9 seq_num 124) dirty exists 2024-04-20T02:52:14.513 INFO:tasks.rados.rados.0.smithi105.stdout:1478: finishing write tid 1 to smithi10548823-6 2024-04-20T02:52:14.513 INFO:tasks.rados.rados.0.smithi105.stdout:1478: finishing write tid 2 to smithi10548823-6 2024-04-20T02:52:14.513 INFO:tasks.rados.rados.0.smithi105.stdout:1466: done (14 left) 2024-04-20T02:52:14.513 INFO:tasks.rados.rados.0.smithi105.stdout:1468: done (13 left) 2024-04-20T02:52:14.513 INFO:tasks.rados.rados.0.smithi105.stdout:1469: done (12 left) 2024-04-20T02:52:14.513 INFO:tasks.rados.rados.0.smithi105.stdout:1470: done (11 left) 2024-04-20T02:52:14.513 INFO:tasks.rados.rados.0.smithi105.stdout:1471: done (10 left) 2024-04-20T02:52:14.513 INFO:tasks.rados.rados.0.smithi105.stdout:1472: done (9 left) 2024-04-20T02:52:14.514 INFO:tasks.rados.rados.0.smithi105.stdout:1474: done (8 left) 2024-04-20T02:52:14.514 INFO:tasks.rados.rados.0.smithi105.stdout:1475: done (7 left) 2024-04-20T02:52:14.514 INFO:tasks.rados.rados.0.smithi105.stdout:1476: done (6 left) 2024-04-20T02:52:14.514 INFO:tasks.rados.rados.0.smithi105.stdout:1477: done (5 left) 2024-04-20T02:52:14.514 INFO:tasks.rados.rados.0.smithi105.stdout:1479: done (4 left) 2024-04-20T02:52:14.514 INFO:tasks.rados.rados.0.smithi105.stdout:1480: snap_remove snap 25 2024-04-20T02:52:14.514 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:14.512+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:14.516 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:52:14.518 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:52:14.529 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-20T02:52:14.207035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":234236915,"num_objects":164,"num_object_clones":14,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":4,"num_read":2249,"num_read_kb":1900968,"num_write":1579,"num_write_kb":424101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":64482439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":31,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":691616,"kb_used_data":441944,"kb_used_omap":12,"kb_used_meta":249523,"kb_avail":754283104,"statfs":{"total":773094113280,"available":772385898496,"internally_reserved":0,"allocated":452550656,"data_stored":476302185,"data_compressed":10849521,"data_compressed_allocated":11599872,"data_compressed_original":27111424,"omap_allocated":12807,"internal_metadata":255512057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9083446,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":58,"num_read_kb":82728,"num_write":27,"num_write_kb":7368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.075188"},"pg_stats":[{"pgid":"4.17","version":"103'87","reported_seq":417,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.446863+0000","last_change":"2024-04-20T02:52:09.132125+0000","last_active":"2024-04-20T02:52:11.446863+0000","last_peered":"2024-04-20T02:52:11.446863+0000","last_clean":"2024-04-20T02:52:11.446863+0000","last_became_active":"2024-04-20T02:52:01.132773+0000","last_became_peered":"2024-04-20T02:52:01.132773+0000","last_unstale":"2024-04-20T02:52:11.446863+0000","last_undegraded":"2024-04-20T02:52:11.446863+0000","last_fullsized":"2024-04-20T02:52:11.446863+0000","mapping_epoch":102,"log_start":"94'78","ondisk_log_start":"94'78","created":96,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'50","last_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.487770+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":131,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019136573,"objects_trimmed":0,"snaptrim_duration":0.00096482399999999998,"stat_sum":{"num_bytes":4201841,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":80892,"num_write":21,"num_write_kb":6419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1378953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"4.1","version":"101'112","reported_seq":426,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.133363+0000","last_change":"2024-04-20T02:52:09.133628+0000","last_active":"2024-04-20T02:52:09.133363+0000","last_peered":"2024-04-20T02:52:09.133363+0000","last_clean":"2024-04-20T02:52:09.133363+0000","last_became_active":"2024-04-20T02:51:18.508043+0000","last_became_peered":"2024-04-20T02:51:18.508043+0000","last_unstale":"2024-04-20T02:52:09.133363+0000","last_undegraded":"2024-04-20T02:52:09.133363+0000","last_fullsized":"2024-04-20T02:52:09.133363+0000","mapping_epoch":46,"log_start":"95'107","ondisk_log_start":"95'107","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'101","last_scrub_stamp":"2024-04-20T02:51:34.098939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:34.098939+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":103,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:56.647060+0000","scrub_duration":0.0019988520000000002,"objects_trimmed":0,"snaptrim_duration":0.00044001000000000001,"stat_sum":{"num_bytes":2218557,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":38564,"num_write":9,"num_write_kb":2627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":4,"reported_epoch":111,"state":"peering","last_fresh":"2024-04-20T02:52:09.134369+0000","last_change":"2024-04-20T02:52:09.134369+0000","last_active":"2024-04-20T02:50:26.619512+0000","last_peered":"2024-04-20T02:50:26.619512+0000","last_clean":"2024-04-20T02:50:26.619512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T02:52:09.134369+0000","last_undegraded":"2024-04-20T02:52:09.134369+0000","last_fullsized":"2024-04-20T02:52:09.134369+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:56.983760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.983760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:06.108963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"114'121","reported_seq":429,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-20T02:52:12.204211+0000","last_change":"2024-04-20T02:52:09.130091+0000","last_active":"2024-04-20T02:52:12.204211+0000","last_peered":"2024-04-20T02:52:12.204211+0000","last_clean":"2024-04-20T02:52:12.204211+0000","last_became_active":"2024-04-20T02:51:18.505708+0000","last_became_peered":"2024-04-20T02:51:18.505708+0000","last_unstale":"2024-04-20T02:52:12.204211+0000","last_undegraded":"2024-04-20T02:52:12.204211+0000","last_fullsized":"2024-04-20T02:52:12.204211+0000","mapping_epoch":22,"log_start":"114'118","ondisk_log_start":"114'118","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.922550+0000","scrub_duration":0.001820667,"objects_trimmed":0,"snaptrim_duration":0.00032542500000000001,"stat_sum":{"num_bytes":13366839,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":76,"num_read_kb":84020,"num_write":37,"num_write_kb":11351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":116,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.446402+0000","last_change":"2024-04-20T02:51:57.514845+0000","last_active":"2024-04-20T02:52:11.446402+0000","last_peered":"2024-04-20T02:52:11.446402+0000","last_clean":"2024-04-20T02:52:11.446402+0000","last_became_active":"2024-04-20T02:50:27.638678+0000","last_became_peered":"2024-04-20T02:50:27.638678+0000","last_unstale":"2024-04-20T02:52:11.446402+0000","last_undegraded":"2024-04-20T02:52:11.446402+0000","last_fullsized":"2024-04-20T02:52:11.446402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:57.514743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:57.514743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:05.344562+0000","scrub_duration":0.001500723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":112,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-20T02:52:08.129061+0000","last_change":"2024-04-20T02:51:41.093997+0000","last_active":"2024-04-20T02:52:08.129061+0000","last_peered":"2024-04-20T02:52:08.129061+0000","last_clean":"2024-04-20T02:52:08.129061+0000","last_became_active":"2024-04-20T02:50:28.091568+0000","last_became_peered":"2024-04-20T02:50:28.091568+0000","last_unstale":"2024-04-20T02:52:08.129061+0000","last_undegraded":"2024-04-20T02:52:08.129061+0000","last_fullsized":"2024-04-20T02:52:08.129061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:41.093905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:41.093905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:01.045646+0000","scrub_duration":0.0019242129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"94'207","reported_seq":547,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.136226+0000","last_change":"2024-04-20T02:52:09.136406+0000","last_active":"2024-04-20T02:52:09.136226+0000","last_peered":"2024-04-20T02:52:09.136226+0000","last_clean":"2024-04-20T02:52:09.136226+0000","last_became_active":"2024-04-20T02:51:18.502004+0000","last_became_peered":"2024-04-20T02:51:18.502004+0000","last_unstale":"2024-04-20T02:52:09.136226+0000","last_undegraded":"2024-04-20T02:52:09.136226+0000","last_fullsized":"2024-04-20T02:52:09.136226+0000","mapping_epoch":47,"log_start":"91'204","ondisk_log_start":"91'204","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'157","last_scrub_stamp":"2024-04-20T02:51:19.377186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.377186+0000","objects_scrubbed":5,"log_size":3,"log_dups_size":200,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:38.408177+0000","scrub_duration":0.0026503899999999999,"objects_trimmed":0,"snaptrim_duration":0.00026263799999999999,"stat_sum":{"num_bytes":3680733,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":98006,"num_write":15,"num_write_kb":4725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"}]},{"pgid":"3.1","version":"111'190","reported_seq":324,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.145997+0000","last_change":"2024-04-20T02:52:05.079320+0000","last_active":"2024-04-20T02:52:09.145997+0000","last_peered":"2024-04-20T02:52:09.145997+0000","last_clean":"2024-04-20T02:52:09.145997+0000","last_became_active":"2024-04-20T02:50:34.238906+0000","last_became_peered":"2024-04-20T02:50:34.238906+0000","last_unstale":"2024-04-20T02:52:09.145997+0000","last_undegraded":"2024-04-20T02:52:09.145997+0000","last_fullsized":"2024-04-20T02:52:09.145997+0000","mapping_epoch":21,"log_start":"106'187","ondisk_log_start":"106'187","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'187","last_scrub_stamp":"2024-04-20T02:52:05.079195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:52:05.079195+0000","objects_scrubbed":23,"log_size":3,"log_dups_size":212,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:31.755702+0000","scrub_duration":0.0039989029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20515570,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":174,"num_read_kb":7850,"num_write":222,"num_write_kb":62878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":4,"reported_epoch":111,"state":"peering","last_fresh":"2024-04-20T02:52:09.134700+0000","last_change":"2024-04-20T02:51:46.231620+0000","last_active":"2024-04-20T02:50:26.619512+0000","last_peered":"2024-04-20T02:50:26.619512+0000","last_clean":"2024-04-20T02:50:26.619512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T02:52:09.134700+0000","last_undegraded":"2024-04-20T02:52:09.134700+0000","last_fullsized":"2024-04-20T02:52:09.134700+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:28.222203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:28.222203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:45.805939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"101'90","reported_seq":355,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.135063+0000","last_change":"2024-04-20T02:52:09.135249+0000","last_active":"2024-04-20T02:52:09.135063+0000","last_peered":"2024-04-20T02:52:09.135063+0000","last_clean":"2024-04-20T02:52:09.135063+0000","last_became_active":"2024-04-20T02:51:18.504889+0000","last_became_peered":"2024-04-20T02:51:18.504889+0000","last_unstale":"2024-04-20T02:52:09.135063+0000","last_undegraded":"2024-04-20T02:52:09.135063+0000","last_fullsized":"2024-04-20T02:52:09.135063+0000","mapping_epoch":22,"log_start":"88'83","ondisk_log_start":"88'83","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'57","last_scrub_stamp":"2024-04-20T02:51:21.386796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:21.386796+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":82,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:27.553181+0000","scrub_duration":0.0018878359999999999,"objects_trimmed":0,"snaptrim_duration":0.00057183699999999995,"stat_sum":{"num_bytes":3764768,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":23972,"num_write":11,"num_write_kb":2792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"}]},{"pgid":"3.2","version":"112'181","reported_seq":399,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.131740+0000","last_change":"2024-04-20T02:51:44.446457+0000","last_active":"2024-04-20T02:52:11.131740+0000","last_peered":"2024-04-20T02:52:11.131740+0000","last_clean":"2024-04-20T02:52:11.131740+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:52:11.131740+0000","last_undegraded":"2024-04-20T02:52:11.131740+0000","last_fullsized":"2024-04-20T02:52:11.131740+0000","mapping_epoch":49,"log_start":"105'178","ondisk_log_start":"105'178","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'130","last_scrub_stamp":"2024-04-20T02:51:44.446332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:51:44.446332+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":186,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:11.809200+0000","scrub_duration":0.0039636890000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28536400,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":155,"num_read_kb":10491,"num_write":209,"num_write_kb":73744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":265,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-20T02:52:08.129159+0000","last_change":"2024-04-20T02:51:42.076681+0000","last_active":"2024-04-20T02:52:08.129159+0000","last_peered":"2024-04-20T02:52:08.129159+0000","last_clean":"2024-04-20T02:52:08.129159+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:52:08.129159+0000","last_undegraded":"2024-04-20T02:52:08.129159+0000","last_fullsized":"2024-04-20T02:52:08.129159+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:51:42.076506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:51:42.076506+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:59.869381+0000","scrub_duration":0.002077785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":93,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-20T02:52:12.144957+0000","last_change":"2024-04-20T02:51:30.923927+0000","last_active":"2024-04-20T02:52:12.144957+0000","last_peered":"2024-04-20T02:52:12.144957+0000","last_clean":"2024-04-20T02:52:12.144957+0000","last_became_active":"2024-04-20T02:51:01.151544+0000","last_became_peered":"2024-04-20T02:51:01.151544+0000","last_unstale":"2024-04-20T02:52:12.144957+0000","last_undegraded":"2024-04-20T02:52:12.144957+0000","last_fullsized":"2024-04-20T02:52:12.144957+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:30.923833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:30.923833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:47.551592+0000","scrub_duration":0.001311373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"56'33","reported_seq":127,"reported_epoch":111,"state":"peering","last_fresh":"2024-04-20T02:52:09.132968+0000","last_change":"2024-04-20T02:52:09.132968+0000","last_active":"2024-04-20T02:52:09.132968+0000","last_peered":"2024-04-20T02:51:10.459057+0000","last_clean":"2024-04-20T02:51:10.459057+0000","last_became_active":"2024-04-20T02:50:35.413590+0000","last_became_peered":"2024-04-20T02:50:35.413590+0000","last_unstale":"2024-04-20T02:52:09.132968+0000","last_undegraded":"2024-04-20T02:52:09.132968+0000","last_fullsized":"2024-04-20T02:52:09.132968+0000","mapping_epoch":111,"log_start":"53'30","ondisk_log_start":"53'30","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'33","last_scrub_stamp":"2024-04-20T02:51:32.144366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.144366+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.00038239699999999998,"stat_sum":{"num_bytes":4739995,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":44474,"num_write":11,"num_write_kb":3858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"}]},{"pgid":"3.0","version":"106'161","reported_seq":273,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.445722+0000","last_change":"2024-04-20T02:50:34.241909+0000","last_active":"2024-04-20T02:52:11.445722+0000","last_peered":"2024-04-20T02:52:11.445722+0000","last_clean":"2024-04-20T02:52:11.445722+0000","last_became_active":"2024-04-20T02:50:34.241613+0000","last_became_peered":"2024-04-20T02:50:34.241613+0000","last_unstale":"2024-04-20T02:52:11.445722+0000","last_undegraded":"2024-04-20T02:52:11.445722+0000","last_fullsized":"2024-04-20T02:52:11.445722+0000","mapping_epoch":21,"log_start":"105'158","ondisk_log_start":"105'158","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:50:33.213730+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":170,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19811027,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":136,"num_read_kb":5655,"num_write":195,"num_write_kb":57402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":114,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.924652+0000","last_change":"2024-04-20T02:51:41.175139+0000","last_active":"2024-04-20T02:52:11.924652+0000","last_peered":"2024-04-20T02:52:11.924652+0000","last_clean":"2024-04-20T02:52:11.924652+0000","last_became_active":"2024-04-20T02:50:28.091253+0000","last_became_peered":"2024-04-20T02:50:28.091253+0000","last_unstale":"2024-04-20T02:52:11.924652+0000","last_undegraded":"2024-04-20T02:52:11.924652+0000","last_fullsized":"2024-04-20T02:52:11.924652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:41.175035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:41.175035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:46.711136+0000","scrub_duration":0.0014447850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"101'87","reported_seq":413,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:12.091750+0000","last_change":"2024-04-20T02:52:09.132287+0000","last_active":"2024-04-20T02:52:12.091750+0000","last_peered":"2024-04-20T02:52:12.091750+0000","last_clean":"2024-04-20T02:52:12.091750+0000","last_became_active":"2024-04-20T02:51:55.289731+0000","last_became_peered":"2024-04-20T02:51:55.289731+0000","last_unstale":"2024-04-20T02:52:12.091750+0000","last_undegraded":"2024-04-20T02:52:12.091750+0000","last_fullsized":"2024-04-20T02:52:12.091750+0000","mapping_epoch":46,"log_start":"94'78","ondisk_log_start":"94'78","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'50","last_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.487770+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":131,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019136573,"objects_trimmed":0,"snaptrim_duration":0.00098967099999999991,"stat_sum":{"num_bytes":4778221,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":71380,"num_write":21,"num_write_kb":6419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1378953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.3","version":"105'165","reported_seq":288,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-04-20T02:52:08.132843+0000","last_change":"2024-04-20T02:52:05.855910+0000","last_active":"2024-04-20T02:52:08.132843+0000","last_peered":"2024-04-20T02:52:08.132843+0000","last_clean":"2024-04-20T02:52:08.132843+0000","last_became_active":"2024-04-20T02:50:34.241638+0000","last_became_peered":"2024-04-20T02:50:34.241638+0000","last_unstale":"2024-04-20T02:52:08.132843+0000","last_undegraded":"2024-04-20T02:52:08.132843+0000","last_fullsized":"2024-04-20T02:52:08.132843+0000","mapping_epoch":21,"log_start":"105'162","ondisk_log_start":"105'162","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'165","last_scrub_stamp":"2024-04-20T02:52:05.855745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:52:05.855745+0000","objects_scrubbed":23,"log_size":3,"log_dups_size":168,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:19.508690+0000","scrub_duration":0.0059089679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23619784,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":142,"num_read_kb":4928,"num_write":195,"num_write_kb":57690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":93,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-20T02:52:12.144807+0000","last_change":"2024-04-20T02:51:52.777359+0000","last_active":"2024-04-20T02:52:12.144807+0000","last_peered":"2024-04-20T02:52:12.144807+0000","last_clean":"2024-04-20T02:52:12.144807+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:52:12.144807+0000","last_undegraded":"2024-04-20T02:52:12.144807+0000","last_fullsized":"2024-04-20T02:52:12.144807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:52.777234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:52.777234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:10.584062+0000","scrub_duration":0.0012962270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"56'17","reported_seq":253,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.445913+0000","last_change":"2024-04-20T02:52:09.131742+0000","last_active":"2024-04-20T02:52:11.445913+0000","last_peered":"2024-04-20T02:52:11.445913+0000","last_clean":"2024-04-20T02:52:11.445913+0000","last_became_active":"2024-04-20T02:51:18.509276+0000","last_became_peered":"2024-04-20T02:51:18.509276+0000","last_unstale":"2024-04-20T02:52:11.445913+0000","last_undegraded":"2024-04-20T02:52:11.445913+0000","last_fullsized":"2024-04-20T02:52:11.445913+0000","mapping_epoch":22,"log_start":"53'14","ondisk_log_start":"53'14","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:51:22.420320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:22.420320+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:37.257627+0000","scrub_duration":0.0016858400000000001,"objects_trimmed":0,"snaptrim_duration":0.001054365,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"4.2","version":"0'0","reported_seq":238,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.446027+0000","last_change":"2024-04-20T02:52:09.131188+0000","last_active":"2024-04-20T02:52:11.446027+0000","last_peered":"2024-04-20T02:52:11.446027+0000","last_clean":"2024-04-20T02:52:11.446027+0000","last_became_active":"2024-04-20T02:51:18.506462+0000","last_became_peered":"2024-04-20T02:51:18.506462+0000","last_unstale":"2024-04-20T02:52:11.446027+0000","last_undegraded":"2024-04-20T02:52:11.446027+0000","last_fullsized":"2024-04-20T02:52:11.446027+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:33.440257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:51:33.440257+0000","last_clean_scrub_stamp":"2024-04-20T02:51:33.440257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:00.147584+0000","scrub_duration":0.001747176,"objects_trimmed":0,"snaptrim_duration":0.0011877229999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.445734+0000","last_change":"2024-04-20T02:52:05.461882+0000","last_active":"2024-04-20T02:52:11.445734+0000","last_peered":"2024-04-20T02:52:11.445734+0000","last_clean":"2024-04-20T02:52:11.445734+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:52:11.445734+0000","last_undegraded":"2024-04-20T02:52:11.445734+0000","last_fullsized":"2024-04-20T02:52:11.445734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:50:26.619512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"115'105","reported_seq":401,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-20T02:52:13.189948+0000","last_change":"2024-04-20T02:52:13.146201+0000","last_active":"2024-04-20T02:52:13.189948+0000","last_peered":"2024-04-20T02:52:13.189948+0000","last_clean":"2024-04-20T02:52:13.189948+0000","last_became_active":"2024-04-20T02:52:13.145577+0000","last_became_peered":"2024-04-20T02:52:13.145577+0000","last_unstale":"2024-04-20T02:52:13.189948+0000","last_undegraded":"2024-04-20T02:52:13.189948+0000","last_fullsized":"2024-04-20T02:52:13.189948+0000","mapping_epoch":62,"log_start":"115'102","ondisk_log_start":"115'102","created":22,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'74","last_scrub_stamp":"2024-04-20T02:51:56.784299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.784299+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":260,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:15.219512+0000","scrub_duration":0.0015920019999999999,"objects_trimmed":0,"snaptrim_duration":0.00091858499999999997,"stat_sum":{"num_bytes":8454502,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":3,"num_read_kb":1,"num_write":11,"num_write_kb":2303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"4.9","version":"111'293","reported_seq":668,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.211618+0000","last_change":"2024-04-20T02:52:09.145582+0000","last_active":"2024-04-20T02:52:09.211618+0000","last_peered":"2024-04-20T02:52:09.211618+0000","last_clean":"2024-04-20T02:52:09.211618+0000","last_became_active":"2024-04-20T02:52:09.143912+0000","last_became_peered":"2024-04-20T02:52:09.143912+0000","last_unstale":"2024-04-20T02:52:09.211618+0000","last_undegraded":"2024-04-20T02:52:09.211618+0000","last_fullsized":"2024-04-20T02:52:09.211618+0000","mapping_epoch":22,"log_start":"105'289","ondisk_log_start":"105'289","created":22,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'280","last_scrub_stamp":"2024-04-20T02:51:56.947305+0000","last_deep_scrub":"98'280","last_deep_scrub_stamp":"2024-04-20T02:51:56.947305+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.947305+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":910,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020870665,"objects_trimmed":0,"snaptrim_duration":0.0011702030000000001,"stat_sum":{"num_bytes":17073504,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":47,"num_read_kb":57038,"num_write":31,"num_write_kb":11358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"}]},{"pgid":"4.a","version":"109'57","reported_seq":321,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-20T02:52:12.152854+0000","last_change":"2024-04-20T02:52:10.138833+0000","last_active":"2024-04-20T02:52:12.152854+0000","last_peered":"2024-04-20T02:52:12.152854+0000","last_clean":"2024-04-20T02:52:12.152854+0000","last_became_active":"2024-04-20T02:52:10.138005+0000","last_became_peered":"2024-04-20T02:52:10.138005+0000","last_unstale":"2024-04-20T02:52:12.152854+0000","last_undegraded":"2024-04-20T02:52:12.152854+0000","last_fullsized":"2024-04-20T02:52:12.152854+0000","mapping_epoch":111,"log_start":"90'53","ondisk_log_start":"90'53","created":22,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'53","last_scrub_stamp":"2024-04-20T02:52:05.810627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:05.810627+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":53,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:13.232408+0000","scrub_duration":0.0022334680000000002,"objects_trimmed":0,"snaptrim_duration":0.00043810300000000002,"stat_sum":{"num_bytes":6267459,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":134,"num_read_kb":153220,"num_write":36,"num_write_kb":9637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"4.b","version":"103'157","reported_seq":433,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.134918+0000","last_change":"2024-04-20T02:52:09.135197+0000","last_active":"2024-04-20T02:52:09.134918+0000","last_peered":"2024-04-20T02:52:09.134918+0000","last_clean":"2024-04-20T02:52:09.134918+0000","last_became_active":"2024-04-20T02:50:35.242331+0000","last_became_peered":"2024-04-20T02:50:35.242331+0000","last_unstale":"2024-04-20T02:52:09.134918+0000","last_undegraded":"2024-04-20T02:52:09.134918+0000","last_fullsized":"2024-04-20T02:52:09.134918+0000","mapping_epoch":22,"log_start":"101'154","ondisk_log_start":"101'154","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'157","last_scrub_stamp":"2024-04-20T02:52:07.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:07.013316+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":154,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:15.985329+0000","scrub_duration":0.002193338,"objects_trimmed":0,"snaptrim_duration":0.00062084900000000003,"stat_sum":{"num_bytes":11496021,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":203,"num_read_kb":247165,"num_write":85,"num_write_kb":22744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"}]},{"pgid":"4.c","version":"107'95","reported_seq":313,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.215473+0000","last_change":"2024-04-20T02:52:09.134476+0000","last_active":"2024-04-20T02:52:09.215473+0000","last_peered":"2024-04-20T02:52:09.215473+0000","last_clean":"2024-04-20T02:52:09.215473+0000","last_became_active":"2024-04-20T02:50:35.241480+0000","last_became_peered":"2024-04-20T02:50:35.241480+0000","last_unstale":"2024-04-20T02:52:09.215473+0000","last_undegraded":"2024-04-20T02:52:09.215473+0000","last_fullsized":"2024-04-20T02:52:09.215473+0000","mapping_epoch":22,"log_start":"95'92","ondisk_log_start":"95'92","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":92,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000967208,"stat_sum":{"num_bytes":7479115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":63538,"num_write":53,"num_write_kb":14819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"}]},{"pgid":"4.d","version":"111'177","reported_seq":473,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.261157+0000","last_change":"2024-04-20T02:52:09.136326+0000","last_active":"2024-04-20T02:52:09.261157+0000","last_peered":"2024-04-20T02:52:09.261157+0000","last_clean":"2024-04-20T02:52:09.261157+0000","last_became_active":"2024-04-20T02:50:35.528107+0000","last_became_peered":"2024-04-20T02:50:35.528107+0000","last_unstale":"2024-04-20T02:52:09.261157+0000","last_undegraded":"2024-04-20T02:52:09.261157+0000","last_fullsized":"2024-04-20T02:52:09.261157+0000","mapping_epoch":22,"log_start":"111'173","ondisk_log_start":"111'173","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'169","last_scrub_stamp":"2024-04-20T02:52:08.841360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:08.841360+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":173,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:22.494159+0000","scrub_duration":0.002314502,"objects_trimmed":0,"snaptrim_duration":0.0015099200000000001,"stat_sum":{"num_bytes":12116067,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":279,"num_read_kb":308677,"num_write":102,"num_write_kb":27012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"}]},{"pgid":"4.e","version":"104'85","reported_seq":318,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.133848+0000","last_change":"2024-04-20T02:52:09.134111+0000","last_active":"2024-04-20T02:52:09.133848+0000","last_peered":"2024-04-20T02:52:09.133848+0000","last_clean":"2024-04-20T02:52:09.133848+0000","last_became_active":"2024-04-20T02:50:35.244872+0000","last_became_peered":"2024-04-20T02:50:35.244872+0000","last_unstale":"2024-04-20T02:52:09.133848+0000","last_undegraded":"2024-04-20T02:52:09.133848+0000","last_fullsized":"2024-04-20T02:52:09.133848+0000","mapping_epoch":22,"log_start":"92'82","ondisk_log_start":"92'82","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'77","last_scrub_stamp":"2024-04-20T02:51:46.088382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:46.088382+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":82,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:57.901141+0000","scrub_duration":0.002170693,"objects_trimmed":0,"snaptrim_duration":0.00060881200000000003,"stat_sum":{"num_bytes":7008519,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":80719,"num_write":41,"num_write_kb":13592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"}]},{"pgid":"4.f","version":"112'31","reported_seq":253,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.446601+0000","last_change":"2024-04-20T02:52:09.132434+0000","last_active":"2024-04-20T02:52:11.446601+0000","last_peered":"2024-04-20T02:52:11.446601+0000","last_clean":"2024-04-20T02:52:11.446601+0000","last_became_active":"2024-04-20T02:50:35.245323+0000","last_became_peered":"2024-04-20T02:50:35.245323+0000","last_unstale":"2024-04-20T02:52:11.446601+0000","last_undegraded":"2024-04-20T02:52:11.446601+0000","last_fullsized":"2024-04-20T02:52:11.446601+0000","mapping_epoch":22,"log_start":"103'27","ondisk_log_start":"103'27","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":27,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001039611,"stat_sum":{"num_bytes":4647964,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":61,"num_read_kb":81750,"num_write":23,"num_write_kb":6661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"4.10","version":"114'107","reported_seq":482,"reported_epoch":114,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:52:12.154022+0000","last_change":"2024-04-20T02:52:02.137807+0000","last_active":"2024-04-20T02:52:12.154022+0000","last_peered":"2024-04-20T02:52:12.154022+0000","last_clean":"2024-04-20T02:51:59.302382+0000","last_became_active":"2024-04-20T02:52:02.136888+0000","last_became_peered":"2024-04-20T02:52:02.136888+0000","last_unstale":"2024-04-20T02:52:12.154022+0000","last_undegraded":"2024-04-20T02:52:12.154022+0000","last_fullsized":"2024-04-20T02:52:12.154022+0000","mapping_epoch":103,"log_start":"91'103","ondisk_log_start":"91'103","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":5,"last_scrub":"70'78","last_scrub_stamp":"2024-04-20T02:51:32.278907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:32.278907+0000","objects_scrubbed":5,"log_size":4,"log_dups_size":101,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:52.494873+0000","scrub_duration":0.0025619089999999998,"objects_trimmed":0,"snaptrim_duration":0.001810437,"stat_sum":{"num_bytes":8357135,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":109144,"num_write":11,"num_write_kb":2892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4056495,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,1,6],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"111'179","reported_seq":636,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.209441+0000","last_change":"2024-04-20T02:52:09.134838+0000","last_active":"2024-04-20T02:52:09.209441+0000","last_peered":"2024-04-20T02:52:09.209441+0000","last_clean":"2024-04-20T02:52:09.209441+0000","last_became_active":"2024-04-20T02:52:04.132446+0000","last_became_peered":"2024-04-20T02:52:04.132446+0000","last_unstale":"2024-04-20T02:52:09.209441+0000","last_undegraded":"2024-04-20T02:52:09.209441+0000","last_fullsized":"2024-04-20T02:52:09.209441+0000","mapping_epoch":105,"log_start":"107'175","ondisk_log_start":"107'175","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'147","last_scrub_stamp":"2024-04-20T02:51:44.085213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:44.085213+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":175,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:12.335888+0000","scrub_duration":0.0022866549999999999,"objects_trimmed":0,"snaptrim_duration":0.001790069,"stat_sum":{"num_bytes":7881900,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":25555,"num_write":17,"num_write_kb":3137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":14392198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"}]},{"pgid":"4.12","version":"0'0","reported_seq":274,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.445769+0000","last_change":"2024-04-20T02:52:09.130969+0000","last_active":"2024-04-20T02:52:11.445769+0000","last_peered":"2024-04-20T02:52:11.445769+0000","last_clean":"2024-04-20T02:52:11.445769+0000","last_became_active":"2024-04-20T02:52:01.135950+0000","last_became_peered":"2024-04-20T02:52:01.135950+0000","last_unstale":"2024-04-20T02:52:11.445769+0000","last_undegraded":"2024-04-20T02:52:11.445769+0000","last_fullsized":"2024-04-20T02:52:11.445769+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:45.458449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:45.458449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:50.931380+0000","scrub_duration":0.0013346709999999999,"objects_trimmed":0,"snaptrim_duration":0.0011801120000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"4.13","version":"112'243","reported_seq":676,"reported_epoch":113,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:52:11.446277+0000","last_change":"2024-04-20T02:52:02.153024+0000","last_active":"2024-04-20T02:52:11.446277+0000","last_peered":"2024-04-20T02:52:11.446277+0000","last_clean":"2024-04-20T02:51:59.287088+0000","last_became_active":"2024-04-20T02:52:02.142421+0000","last_became_peered":"2024-04-20T02:52:02.142421+0000","last_unstale":"2024-04-20T02:52:11.446277+0000","last_undegraded":"2024-04-20T02:52:11.446277+0000","last_fullsized":"2024-04-20T02:52:11.446277+0000","mapping_epoch":103,"log_start":"105'240","ondisk_log_start":"105'240","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":5,"last_scrub":"73'167","last_scrub_stamp":"2024-04-20T02:51:33.169112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:33.169112+0000","objects_scrubbed":10,"log_size":3,"log_dups_size":240,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:52:48.762621+0000","scrub_duration":0.015460714,"objects_trimmed":1,"snaptrim_duration":0.00057284800000000004,"stat_sum":{"num_bytes":7731286,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":153208,"num_write":20,"num_write_kb":5084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8833476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"104'35","reported_seq":235,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-20T02:52:11.445952+0000","last_change":"2024-04-20T02:52:09.131293+0000","last_active":"2024-04-20T02:52:11.445952+0000","last_peered":"2024-04-20T02:52:11.445952+0000","last_clean":"2024-04-20T02:52:11.445952+0000","last_became_active":"2024-04-20T02:52:04.133340+0000","last_became_peered":"2024-04-20T02:52:04.133340+0000","last_unstale":"2024-04-20T02:52:11.445952+0000","last_undegraded":"2024-04-20T02:52:11.445952+0000","last_fullsized":"2024-04-20T02:52:11.445952+0000","mapping_epoch":105,"log_start":"93'29","ondisk_log_start":"93'29","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":29,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080000600000000002,"stat_sum":{"num_bytes":487453,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":26722,"num_write":14,"num_write_kb":3607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3776971,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"4.15","version":"104'65","reported_seq":338,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-04-20T02:52:06.574513+0000","last_change":"2024-04-20T02:52:02.133521+0000","last_active":"2024-04-20T02:52:06.574513+0000","last_peered":"2024-04-20T02:52:06.574513+0000","last_clean":"2024-04-20T02:52:06.574513+0000","last_became_active":"2024-04-20T02:52:02.130111+0000","last_became_peered":"2024-04-20T02:52:02.130111+0000","last_unstale":"2024-04-20T02:52:06.574513+0000","last_undegraded":"2024-04-20T02:52:06.574513+0000","last_fullsized":"2024-04-20T02:52:06.574513+0000","mapping_epoch":103,"log_start":"93'62","ondisk_log_start":"93'62","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002596508,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":93,"num_read_kb":123899,"num_write":30,"num_write_kb":8934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":16438198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c","length":"1"}]},{"pgid":"4.16","version":"101'85","reported_seq":466,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-04-20T02:52:09.134789+0000","last_change":"2024-04-20T02:52:09.135014+0000","last_active":"2024-04-20T02:52:09.134789+0000","last_peered":"2024-04-20T02:52:09.134789+0000","last_clean":"2024-04-20T02:52:09.134789+0000","last_became_active":"2024-04-20T02:52:04.136440+0000","last_became_peered":"2024-04-20T02:52:04.136440+0000","last_unstale":"2024-04-20T02:52:09.134789+0000","last_undegraded":"2024-04-20T02:52:09.134789+0000","last_fullsized":"2024-04-20T02:52:09.134789+0000","mapping_epoch":105,"log_start":"97'80","ondisk_log_start":"97'80","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'85","last_scrub_stamp":"2024-04-20T02:52:08.050825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:08.050825+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":78,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:34.017551+0000","scrub_duration":0.0020325339999999999,"objects_trimmed":0,"snaptrim_duration":0.00049687699999999995,"stat_sum":{"num_bytes":5411887,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"1"},{"start":"30","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":24,"stat_sum":{"num_bytes":141163766,"num_objects":57,"num_object_clones":14,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":4,"num_read":1526,"num_read_kb":1871944,"num_write":599,"num_write_kb":169971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":50255244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":31,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285057024,"data_stored":306742806,"data_compressed":1118431,"data_compressed_allocated":983040,"data_compressed_original":2334720,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":92482781,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":607,"num_read_kb":28924,"num_write":821,"num_write_kb":251714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":168366080,"data_stored":181027474,"data_compressed":9999143,"data_compressed_allocated":10752000,"data_compressed_original":23764992,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509468,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50112,"kb_used_data":18988,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94321728,"statfs":{"total":96636764160,"available":96585449472,"internally_reserved":0,"allocated":19443712,"data_stored":20028656,"data_compressed":79905,"data_compressed_allocated":110592,"data_compressed_original":643072,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.56699999999999995}]},{"osd":1,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.437}]},{"osd":3,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.502},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.48699999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":0.57699999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":0.59299999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757265,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67572,"kb_used_data":35744,"kb_used_omap":1,"kb_used_meta":31806,"kb_avail":94304268,"statfs":{"total":96636764160,"available":96567570432,"internally_reserved":0,"allocated":36601856,"data_stored":38581503,"data_compressed":1051491,"data_compressed_allocated":1122304,"data_compressed_original":2363392,"omap_allocated":1657,"internal_metadata":32569735},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.40000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.374}]},{"osd":2,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.35799999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.39200000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.32100000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.439},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.443}]},{"osd":7,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.45700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56848,"kb_used_data":29820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94314992,"statfs":{"total":96636764160,"available":96578551808,"internally_reserved":0,"allocated":30535680,"data_stored":33314106,"data_compressed":824870,"data_compressed_allocated":884736,"data_compressed_original":1953792,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.35499999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.317}]},{"osd":3,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.376},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":11.877000000000001,"5min":11.877000000000001,"15min":11.877000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.46700000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.2450000000000001,"5min":2.2450000000000001,"15min":2.2450000000000001},"last":0.495}]},{"osd":6,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":11.805,"5min":11.805,"15min":11.805},"last":0.51600000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:51:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.439}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111344,"kb_used_data":80228,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94260496,"statfs":{"total":96636764160,"available":96522747904,"internally_reserved":0,"allocated":82153472,"data_stored":87791341,"data_compressed":1031758,"data_compressed_allocated":1036288,"data_compressed_original":2478080,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.24199999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.373}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.32800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.312},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.40300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.443}]},{"osd":7,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.41899999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94680,"kb_used_data":63560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94277160,"statfs":{"total":96636764160,"available":96539811840,"internally_reserved":0,"allocated":65085440,"data_stored":68924577,"data_compressed":1871589,"data_compressed_allocated":1994752,"data_compressed_original":4440064,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.435},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.30399999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.54900000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:51:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.49299999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.57099999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111868,"kb_used_data":80748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94259972,"statfs":{"total":96636764160,"available":96522211328,"internally_reserved":0,"allocated":82685952,"data_stored":88343725,"data_compressed":2740866,"data_compressed_allocated":2920448,"data_compressed_original":6946816,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":9.9900000000000002,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.53600000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.501},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.0369999999999999,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.495}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.51800000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.0099999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":9.9570000000000007,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.45800000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.8679999999999999,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.377},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.48599999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.436}]},{"osd":7,"last update":"Sat Apr 20 02:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.98,"5min":3.98,"15min":3.98},"last":0.47099999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509464,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85804,"kb_used_data":54680,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94286036,"statfs":{"total":96636764160,"available":96548900864,"internally_reserved":0,"allocated":55992320,"data_stored":57453553,"data_compressed":1585526,"data_compressed_allocated":1736704,"data_compressed_original":4300800,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.439},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.375}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.42399999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.40600000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.58199999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.63}]},{"osd":6,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67900000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.1449999999999996,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.61399999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509466,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113388,"kb_used_data":78176,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94258452,"statfs":{"total":96636764160,"available":96520654848,"internally_reserved":0,"allocated":80052224,"data_stored":81864724,"data_compressed":1663516,"data_compressed_allocated":1794048,"data_compressed_original":3985408,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.44800000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.36699999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.42499999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.497},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.59099999999999997}]},{"osd":5,"last update":"Sat Apr 20 02:51:55 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.51000000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:51:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26542080,"data_stored":28510164,"data_compressed":1509523,"data_compressed_allocated":1630208,"data_compressed_original":3657728,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12550144,"data_stored":13573233,"data_compressed":734140,"data_compressed_allocated":790528,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12550144,"data_stored":13573233,"data_compressed":734140,"data_compressed_allocated":790528,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28049408,"data_stored":30161485,"data_compressed":1741101,"data_compressed_allocated":1867776,"data_compressed_original":4046848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":26542080,"data_stored":28510164,"data_compressed":1509523,"data_compressed_allocated":1630208,"data_compressed_original":3657728,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16556032,"data_stored":17762113,"data_compressed":1051491,"data_compressed_allocated":1122304,"data_compressed_original":2363392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45576192,"data_stored":48937082,"data_compressed":2719225,"data_compressed_allocated":2920448,"data_compressed_original":6369280,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28119040,"data_stored":27855845,"data_compressed":70685,"data_compressed_allocated":73728,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68276224,"data_stored":73636728,"data_compressed":297618,"data_compressed_allocated":245760,"data_compressed_original":643072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19124224,"data_stored":22776250,"data_compressed":145100,"data_compressed_allocated":94208,"data_compressed_original":184320,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":35708928,"data_stored":38181712,"data_compressed":130488,"data_compressed_allocated":126976,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":54603776,"data_stored":58633875,"data_compressed":258423,"data_compressed_allocated":274432,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":21311488,"data_stored":23794071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":38608896,"data_stored":41742883,"data_compressed":141530,"data_compressed_allocated":90112,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":19304448,"data_stored":20121442,"data_compressed":74587,"data_compressed_allocated":77824,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:52:14.613 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:14.613+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:14.657 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:14.658 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:14.658 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:14.658 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:14.668 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:14.714 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:14.714+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:14.814 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:14.815+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:14.838 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:14.838 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:14.838 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:14.839 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:14.839 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:14.849 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:14.915 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:14.916+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:14.979 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.035531058000004, 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.981 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:14.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:52:14.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:52:14.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:52:14.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:52:14.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:52:14.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:14.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:52:14.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.028052740000007, 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:52:14.987 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:14.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:52:14.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:52:14.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:52:14.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:52:14.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.023294840999995, 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:52:14.993 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.599865906999995, 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:14.998 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:14.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.599829024000002, 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:15.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:52:15.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:52:15.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:52:15.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:52:15.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.599804391000006, 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:52:15.011 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.593310979999998, 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.016 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:15.022 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.587246957999994, 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:52:15.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:52:15.028 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.577185193000005, 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:52:15.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.030 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:15.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:15.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:52:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:52:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.576492588999997, 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:52:15.038 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:52:15.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:52:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:52:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:52:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:52:15.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:15.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 80.576480935000006, 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:52:15.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 74.870416379000005, 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:52:15.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:52:15.053 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 74.851355663000007, 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.054 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:52:15.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:15.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 74.845409348999993, 2024-04-20T02:52:15.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:52:15.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:52:15.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.066 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 74.844843908000001, 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.067 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:15.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:15.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 74.844817876999997, 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.073 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:15.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:52:15.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:52:15.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.842605076000002, 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:52:15.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:52:15.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:52:15.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:15.082 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.837255551999998, 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:52:15.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:52:15.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:52:15.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.836785594999998, 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.087 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:15.091 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.836763058999999, 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:52:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.096 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.096 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:15.096 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:15.097 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:15.037+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:15.137 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:15.138+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:15.163 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:1480: done (4 left) 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:1481: read oid 24 snap -1 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:1481: expect (ObjNum 234 snap 22 seq_num 234) 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:1482: write oid 11 current snap is 27 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:1478: finishing write tid 3 to smithi10548823-6 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:1478: finishing write tid 4 to smithi10548823-6 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:1478: oid 6 updating version 0 to 168 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:1478: oid 6 updating version 168 to 169 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:1478: oid 6 version 169 is already newer than 167 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 169 (ObjNum 281 snap 26 seq_num 281) dirty exists 2024-04-20T02:52:15.210 INFO:tasks.rados.rados.0.smithi105.stdout:1478: left oid 6 (ObjNum 281 snap 26 seq_num 281) 2024-04-20T02:52:15.211 INFO:tasks.rados.rados.0.smithi105.stdout:1482: seq_num 282 ranges {739574=725279,1968628=721093} 2024-04-20T02:52:15.221 INFO:tasks.rados.rados.0.smithi105.stdout:1482: writing smithi10548823-11 from 739574 to 1464853 tid 1 2024-04-20T02:52:15.228 INFO:tasks.rados.rados.0.smithi105.stdout:1482: writing smithi10548823-11 from 1968628 to 2689721 tid 2 2024-04-20T02:52:15.228 INFO:tasks.rados.rados.0.smithi105.stdout:1439: done (5 left) 2024-04-20T02:52:15.228 INFO:tasks.rados.rados.0.smithi105.stdout:1478: done (4 left) 2024-04-20T02:52:15.228 INFO:tasks.rados.rados.0.smithi105.stdout:1483: read oid 48 snap -1 2024-04-20T02:52:15.228 INFO:tasks.rados.rados.0.smithi105.stdout:1483: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:15.228 INFO:tasks.rados.rados.0.smithi105.stdout:1484: read oid 30 snap -1 2024-04-20T02:52:15.228 INFO:tasks.rados.rados.0.smithi105.stdout:1484: expect (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:52:15.228 INFO:tasks.rados.rados.0.smithi105.stdout:1485: set_chunk oid 44 target oid 2024-04-20T02:52:15.233 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:15.233 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:15.233 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:15.233 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:15.238 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:15.239+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:15.246 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:15.269 INFO:tasks.rados.rados.0.smithi105.stdout:1485: set_chunk oid 44 offset: 2234368 length: 577024 target oid 7d7b0eb9ec942463e3b18218bd8ad523853076957f42da081773acd6ccb9501e offset: 0 snap 23 2024-04-20T02:52:15.269 INFO:tasks.rados.rados.0.smithi105.stdout:1486: tier_flush oid 40 2024-04-20T02:52:15.269 INFO:tasks.rados.rados.0.smithi105.stdout:1486: tier_flush oid 40 snap -1 2024-04-20T02:52:15.269 INFO:tasks.rados.rados.0.smithi105.stdout:1467: finishing rollback tid 0 to smithi10548823-47 2024-04-20T02:52:15.269 INFO:tasks.rados.rados.0.smithi105.stdout:1487: write (excl) oid 39 current snap is 27 2024-04-20T02:52:15.269 INFO:tasks.rados.rados.0.smithi105.stdout:1467: finishing rollback tid 1 to smithi10548823-47 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 101 (ObjNum 193 snap 0 seq_num 59787856) dirty dne 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:1473: finishing write tid 1 to smithi10548823-8 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:1473: finishing write tid 2 to smithi10548823-8 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:1473: finishing write tid 3 to smithi10548823-8 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:1473: finishing write tid 4 to smithi10548823-8 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:1473: oid 8 updating version 0 to 104 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:1473: oid 8 version 104 is already newer than 103 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:1473: oid 8 updating version 104 to 105 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 105 (ObjNum 280 snap 26 seq_num 280) dirty exists 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:1473: left oid 8 (ObjNum 280 snap 26 seq_num 280) 2024-04-20T02:52:15.270 INFO:tasks.rados.rados.0.smithi105.stdout:1487: seq_num 283 ranges {756381=400528,1910757=464751} 2024-04-20T02:52:15.275 INFO:tasks.rados.rados.0.smithi105.stdout:1487: writing smithi10548823-39 from 756381 to 1156909 tid 1 2024-04-20T02:52:15.283 INFO:tasks.rados.rados.0.smithi105.stdout:1487: writing smithi10548823-39 from 1910757 to 2375508 tid 2 2024-04-20T02:52:15.283 INFO:tasks.rados.rados.0.smithi105.stdout:1467: done (8 left) 2024-04-20T02:52:15.283 INFO:tasks.rados.rados.0.smithi105.stdout:1473: done (7 left) 2024-04-20T02:52:15.283 INFO:tasks.rados.rados.0.smithi105.stdout:1488: read oid 47 snap -1 2024-04-20T02:52:15.283 INFO:tasks.rados.rados.0.smithi105.stdout:1488: expect deleted 2024-04-20T02:52:15.283 INFO:tasks.rados.rados.0.smithi105.stdout:1489: snap_create 2024-04-20T02:52:15.291 INFO:tasks.rados.rados.0.smithi105.stdout:1483: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:15.340 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:15.340+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:15.418 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:15.418 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:15.418 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:15.418 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:15.418 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:15.428 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:15.469 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:15.470+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:15.510 INFO:tasks.rados.rados.0.smithi105.stdout:1481: expect (ObjNum 234 snap 22 seq_num 234) 2024-04-20T02:52:15.541 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:15.542+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:15.599 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:15.599 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:15.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:15.599 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.863575237000006, 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:15.605 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.832286175999997, 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.827595720000005, 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.611 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:52:15.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:52:15.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.820839250999995, 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.617 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:52:15.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:52:15.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:52:15.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:52:15.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.622 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.811023739999996, 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:15.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:52:15.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:52:15.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:52:15.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:52:15.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.809979807999994, 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:52:15.628 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:15.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.809958648000006, 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:15.634 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.638 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.602388067000007, 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:52:15.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:52:15.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:52:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.588193422000003, 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:15.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:52:15.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:52:15.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:52:15.646 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.582226164999994, 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:15.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:52:15.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:52:15.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:52:15.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.573820162000004, 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:15.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:52:15.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:52:15.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:52:15.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.470188121000007, 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:52:15.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:52:15.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:52:15.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.239591810999997, 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:15.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:52:15.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:52:15.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:52:15.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.212647661999995, 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:52:15.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:52:15.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.187289394999993, 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.159148951000006, 2024-04-20T02:52:15.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:52:15.675 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.121780345999994, 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:52:15.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:52:15.679 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.121483423000001, 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:52:15.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:52:15.683 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.116400304999999, 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:52:15.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.687 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.089397746000003, 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:15.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:52:15.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:52:15.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:15.691 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:15.692 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:15.693 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:15.693 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:15.694+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:15.793 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:15.793+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:1482: finishing write tid 1 to smithi10548823-11 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:1482: finishing write tid 2 to smithi10548823-11 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:1482: finishing write tid 3 to smithi10548823-11 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:1482: finishing write tid 4 to smithi10548823-11 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:1482: oid 11 updating version 0 to 129 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:1482: oid 11 version 129 is already newer than 127 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:1482: oid 11 version 129 is already newer than 128 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 129 (ObjNum 282 snap 27 seq_num 282) dirty exists 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:1482: left oid 11 (ObjNum 282 snap 27 seq_num 282) 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:1485: finishing set_chunk to oid 44 2024-04-20T02:52:15.811 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 44 set_chunk returned error code -95 offset: 2234368 length: 577024 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 208 snap 19 seq_num 208) dirty exists 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:1486: got (0) Success 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 0 (ObjNum 248 snap 22 seq_num 248) dirty exists 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:1487: finishing write tid 1 to smithi10548823-39 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:1487: finishing write tid 2 to smithi10548823-39 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:1487: finishing write tid 3 to smithi10548823-39 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:1487: finishing write tid 4 to smithi10548823-39 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:1487: oid 39 updating version 0 to 132 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:1487: oid 39 updating version 132 to 133 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:1487: oid 39 version 133 is already newer than 131 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 133 (ObjNum 283 snap 27 seq_num 283) dirty exists 2024-04-20T02:52:15.812 INFO:tasks.rados.rados.0.smithi105.stdout:1487: left oid 39 (ObjNum 283 snap 27 seq_num 283) 2024-04-20T02:52:15.862 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:15.862 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:15.862 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:15.862 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:15.872 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:15.894 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:15.894+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:15.995 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:15.995+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:16.000 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:16.001 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:16.011 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:16.096 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:16.097+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:16.130 INFO:tasks.rados.rados.0.smithi105.stdout:1481: done (8 left) 2024-04-20T02:52:16.130 INFO:tasks.rados.rados.0.smithi105.stdout:1482: done (7 left) 2024-04-20T02:52:16.130 INFO:tasks.rados.rados.0.smithi105.stdout:1483: done (6 left) 2024-04-20T02:52:16.130 INFO:tasks.rados.rados.0.smithi105.stdout:1485: done (5 left) 2024-04-20T02:52:16.130 INFO:tasks.rados.rados.0.smithi105.stdout:1486: done (4 left) 2024-04-20T02:52:16.130 INFO:tasks.rados.rados.0.smithi105.stdout:1487: done (3 left) 2024-04-20T02:52:16.131 INFO:tasks.rados.rados.0.smithi105.stdout:1488: done (2 left) 2024-04-20T02:52:16.131 INFO:tasks.rados.rados.0.smithi105.stdout:1489: done (1 left) 2024-04-20T02:52:16.131 INFO:tasks.rados.rados.0.smithi105.stdout:1490: rollback oid 41 current snap is 28 2024-04-20T02:52:16.131 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 41 to 27 2024-04-20T02:52:16.131 INFO:tasks.rados.rados.0.smithi105.stdout:1491: snap_create 2024-04-20T02:52:16.133 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:52:16.135 INFO:tasks.rados.rados.0.smithi105.stdout:1490: finishing rollback tid 1 to smithi10548823-41 2024-04-20T02:52:16.135 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 61336744) dirty dne 2024-04-20T02:52:16.157 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 116.579405859, 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:16.184 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:52:16.185 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:52:16.185 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:52:16.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.185 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:16.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.815539459999997, 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:52:16.191 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.192 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.192 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:16.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.195 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.702236419000002, 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.196 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:16.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:16.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:52:16.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:52:16.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:16.202 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.696470902000002, 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:16.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:52:16.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:52:16.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:52:16.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:52:16.209 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.209 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.209 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.209 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:52:16.209 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:16.209 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.342450767000003, 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:16.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:52:16.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.213 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.332087161999993, 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:16.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:52:16.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.324002683000003, 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.218 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:16.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:52:16.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:52:16.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:52:16.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:52:16.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:52:16.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:52:16.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:52:16.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:52:16.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:52:16.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:52:16.224 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.224 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.224 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.224 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:16.224 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:16.224 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.042444458000006, 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:16.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:16.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:52:16.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:16.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:52:16.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:52:16.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:16.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:52:16.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:52:16.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:16.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.037307347999999, 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.232 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:16.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:52:16.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:52:16.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:52:16.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:52:16.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:16.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:52:16.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:52:16.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.037034336000005, 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:16.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:16.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.244 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 76.037004951, 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:52:16.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:16.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:16.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:52:16.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:52:16.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:52:16.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:52:16.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:52:16.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:52:16.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.250 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.251 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.251 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:52:16.251 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:16.251 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 65.728110986000004, 2024-04-20T02:52:16.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035060916999999997, 2024-04-20T02:52:16.251 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.251 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456603+0000", 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001394295 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456616+0000", 2024-04-20T02:52:16.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2605999999999999e-05 2024-04-20T02:52:16.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456631+0000", 2024-04-20T02:52:16.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4735e-05 2024-04-20T02:52:16.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456682+0000", 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1048000000000003e-05 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456705+0000", 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2881e-05 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490235+0000", 2024-04-20T02:52:16.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033530250999999997 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490270+0000", 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5379000000000001e-05 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:52:16.256 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:16.257 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 53.349677651999997, 2024-04-20T02:52:16.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:52:16.257 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.257 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.257 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:52:16.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:52:16.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:52:16.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:52:16.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:52:16.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:52:16.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:52:16.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:52:16.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:52:16.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:52:16.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2132 4.10 e70/63 4:0b0a7da5:::smithi10548823-43:head v 70'76, mlcod=65'73)", 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 48.439079612999997, 2024-04-20T02:52:16.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036823724000000002, 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744239+0000", 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744631+0000", 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039175199999999999 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744644+0000", 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3079e-05 2024-04-20T02:52:16.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744653+0000", 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8629999999999997e-06 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744706+0000", 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2627999999999998e-05 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744737+0000", 2024-04-20T02:52:16.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1387999999999998e-05 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781039+0000", 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036302147 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781064+0000", 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5182999999999999e-05 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2293 4.3 e76/60 4:c07c6daa:::smithi10548823-7:head v 76'165, mlcod=64'161)", 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.994211589999999, 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034106808000000002, 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:16.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189108+0000", 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190758+0000", 2024-04-20T02:52:16.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016504740000000001 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190778+0000", 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9507000000000001e-05 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190787+0000", 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1740000000000006e-06 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190836+0000", 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9351999999999998e-05 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190854+0000", 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7629e-05 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.223165+0000", 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032310885999999997 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.223216+0000", 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0615000000000002e-05 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.902197430000001, 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.271 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:16.275 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.884928995999999, 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.14 98/97 [PushOp(4:2e88ea27:::smithi10548823-12:head, version: 98'33, data_included: [406769~907695,1439434~140881], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:2e88ea27:::smithi10548823-12:head@98'33, size: 2033680, copy_subset: [0~2033680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1580315, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:52:16.279 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.933162161999999, 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033699081999999998, 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251322+0000", 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001164251 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251340+0000", 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.789e-05 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251356+0000", 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5962000000000001e-05 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251428+0000", 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2750000000000007e-05 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251444+0000", 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5248e-05 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.283857+0000", 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032413659999999997 2024-04-20T02:52:16.283 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.284 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.284 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.284 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.6.0:82 4.15 e104/103 4:a9897a5e:::smithi10548823-42:head v 104'64, mlcod=93'62)", 2024-04-20T02:52:16.284 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:16.284 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 14.047447868000001, 2024-04-20T02:52:16.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035573920000000002, 2024-04-20T02:52:16.284 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.284 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135871+0000", 2024-04-20T02:52:16.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:16.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136395+0000", 2024-04-20T02:52:16.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00052370100000000001 2024-04-20T02:52:16.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136409+0000", 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4191e-05 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136418+0000", 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0100000000000001e-06 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136468+0000", 2024-04-20T02:52:16.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9458e-05 2024-04-20T02:52:16.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136499+0000", 2024-04-20T02:52:16.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1062999999999997e-05 2024-04-20T02:52:16.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.171417+0000", 2024-04-20T02:52:16.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034918607999999997 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.171446+0000", 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9000999999999999e-05 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:52:16.289 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.7117832010000003, 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:16.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:52:16.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:16.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:52:16.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:52:16.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:16.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:16.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:16.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:52:16.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:52:16.294 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.294 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.295 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.295 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:16.295 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:16.295 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:16.297 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:16.298 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:16.296+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:16.397 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:16.398+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:16.427 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:16.427 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:16.427 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:16.428 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:16.438 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:16.441 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:16.461 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:16.479 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:16.498 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:16.499 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:16.499+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:16.518 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:16.537 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:16.556 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:16.568 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:16.569 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:16.574 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:16.575 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:16.576 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:16.579 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:16.622 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:16.623+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:16.700 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:16.700+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:16.707 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:16.707 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:16.717 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:16.801 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:16.801+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:16.847 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:16.847 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:16.858 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:16.942 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:16.942+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:17.003 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:17.003+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:17.026 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:17.026 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:17.036 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:17.122 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:17.122+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:17.135 INFO:tasks.rados.rados.0.smithi105.stdout:1490: done (2 left) 2024-04-20T02:52:17.135 INFO:tasks.rados.rados.0.smithi105.stdout:1491: done (1 left) 2024-04-20T02:52:17.135 INFO:tasks.rados.rados.0.smithi105.stdout:1492: write oid 24 current snap is 29 2024-04-20T02:52:17.135 INFO:tasks.rados.rados.0.smithi105.stdout:1492: seq_num 284 ranges {773188=475777,1969239=724762} 2024-04-20T02:52:17.142 INFO:tasks.rados.rados.0.smithi105.stdout:1492: writing smithi10548823-24 from 773188 to 1248965 tid 1 2024-04-20T02:52:17.152 INFO:tasks.rados.rados.0.smithi105.stdout:1492: writing smithi10548823-24 from 1969239 to 2694001 tid 2 2024-04-20T02:52:17.152 INFO:tasks.rados.rados.0.smithi105.stdout:1493: tier_promote oid 3 2024-04-20T02:52:17.152 INFO:tasks.rados.rados.0.smithi105.stdout:1494: read oid 41 snap 22 2024-04-20T02:52:17.152 INFO:tasks.rados.rados.0.smithi105.stdout:1494: expect deleted 2024-04-20T02:52:17.152 INFO:tasks.rados.rados.0.smithi105.stdout:1495: set_chunk oid 13 target oid 2024-04-20T02:52:17.205 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:17.205+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:17.215 INFO:tasks.rados.rados.0.smithi105.stdout:1495: set_chunk oid 13 offset: 1054720 length: 1424896 target oid 802210f544165dfeeb545e0be28befc3e0a27c18094a42f52acfdf0a774917c7 offset: 0 snap -1 2024-04-20T02:52:17.216 INFO:tasks.rados.rados.0.smithi105.stdout:1496: read oid 19 snap 28 2024-04-20T02:52:17.216 INFO:tasks.rados.rados.0.smithi105.stdout:1496: expect (ObjNum 278 snap 26 seq_num 278) 2024-04-20T02:52:17.216 INFO:tasks.rados.rados.0.smithi105.stdout:1497: read oid 22 snap -1 2024-04-20T02:52:17.216 INFO:tasks.rados.rados.0.smithi105.stdout:1497: expect (ObjNum 273 snap 26 seq_num 273) 2024-04-20T02:52:17.216 INFO:tasks.rados.rados.0.smithi105.stdout:1498: write (excl) oid 50 current snap is 29 2024-04-20T02:52:17.216 INFO:tasks.rados.rados.0.smithi105.stdout:1498: seq_num 285 ranges {789995=551026,2027721=468420} 2024-04-20T02:52:17.222 INFO:tasks.rados.rados.0.smithi105.stdout:1498: writing smithi10548823-50 from 789995 to 1341021 tid 1 2024-04-20T02:52:17.227 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:17.228 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:17.233 INFO:tasks.rados.rados.0.smithi105.stdout:1498: writing smithi10548823-50 from 2027721 to 2496141 tid 2 2024-04-20T02:52:17.233 INFO:tasks.rados.rados.0.smithi105.stdout:1499: read oid 34 snap -1 2024-04-20T02:52:17.233 INFO:tasks.rados.rados.0.smithi105.stdout:1499: expect (ObjNum 258 snap 22 seq_num 258) 2024-04-20T02:52:17.234 INFO:tasks.rados.rados.0.smithi105.stdout:1500: set_chunk oid 42 target oid 2024-04-20T02:52:17.242 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:17.248 INFO:tasks.rados.rados.0.smithi105.stdout:1500: set_chunk oid 42 offset: 1217024 length: 668160 target oid 9e038e1308688b019a2a31b3443677bc7c31ba7a4778ccff8f8ca702844eb95b offset: 0 snap -1 2024-04-20T02:52:17.249 INFO:tasks.rados.rados.0.smithi105.stdout:1501: write oid 31 current snap is 29 2024-04-20T02:52:17.249 INFO:tasks.rados.rados.0.smithi105.stdout:1484: expect (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:52:17.306 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:17.307+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:17.379 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:17.379 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:17.390 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:17.434 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:17.434+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:17.507 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:17.508+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:17.519 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:17.519 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:17.530 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:17.602 INFO:tasks.rados.rados.0.smithi105.stdout:1492: finishing write tid 1 to smithi10548823-24 2024-04-20T02:52:17.602 INFO:tasks.rados.rados.0.smithi105.stdout:1492: finishing write tid 2 to smithi10548823-24 2024-04-20T02:52:17.602 INFO:tasks.rados.rados.0.smithi105.stdout:1492: finishing write tid 3 to smithi10548823-24 2024-04-20T02:52:17.602 INFO:tasks.rados.rados.0.smithi105.stdout:1492: finishing write tid 4 to smithi10548823-24 2024-04-20T02:52:17.602 INFO:tasks.rados.rados.0.smithi105.stdout:1492: oid 24 updating version 0 to 296 2024-04-20T02:52:17.602 INFO:tasks.rados.rados.0.smithi105.stdout:1492: oid 24 updating version 296 to 297 2024-04-20T02:52:17.602 INFO:tasks.rados.rados.0.smithi105.stdout:1492: oid 24 updating version 297 to 298 2024-04-20T02:52:17.602 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 298 (ObjNum 284 snap 29 seq_num 284) dirty exists 2024-04-20T02:52:17.603 INFO:tasks.rados.rados.0.smithi105.stdout:1492: left oid 24 (ObjNum 284 snap 29 seq_num 284) 2024-04-20T02:52:17.603 INFO:tasks.rados.rados.0.smithi105.stdout:1493: got (0) Success 2024-04-20T02:52:17.603 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 282 (ObjNum 250 snap 22 seq_num 250) dirty exists 2024-04-20T02:52:17.603 INFO:tasks.rados.rados.0.smithi105.stdout:1495: finishing set_chunk to oid 13 2024-04-20T02:52:17.603 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 13 set_chunk returned error code -95 offset: 1054720 length: 1424896 2024-04-20T02:52:17.603 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 0 (ObjNum 151 snap 10 seq_num 151) dirty exists 2024-04-20T02:52:17.603 INFO:tasks.rados.rados.0.smithi105.stdout:1497: expect (ObjNum 273 snap 26 seq_num 273) 2024-04-20T02:52:17.608 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:17.608+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:17.700 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:17.700 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:17.709 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:17.710+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:17.710 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:17.774 INFO:tasks.rados.rados.0.smithi105.stdout:1496: expect (ObjNum 278 snap 26 seq_num 278) 2024-04-20T02:52:17.811 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:17.811+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:17.882 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:17.882 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:17.892 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:17.912 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:17.912+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:17.963 INFO:tasks.rados.rados.0.smithi105.stdout:1499: expect (ObjNum 258 snap 22 seq_num 258) 2024-04-20T02:52:18.012 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:18.013+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:18.063 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:18.064 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:18.074 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:18.087 INFO:tasks.rados.rados.0.smithi105.stdout:1500: finishing set_chunk to oid 42 2024-04-20T02:52:18.087 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 42 set_chunk returned error code -95 offset: 1217024 length: 668160 2024-04-20T02:52:18.087 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 0 (ObjNum 270 snap 26 seq_num 270) dirty exists 2024-04-20T02:52:18.087 INFO:tasks.rados.rados.0.smithi105.stdout:1498: finishing write tid 1 to smithi10548823-50 2024-04-20T02:52:18.087 INFO:tasks.rados.rados.0.smithi105.stdout:1498: finishing write tid 2 to smithi10548823-50 2024-04-20T02:52:18.087 INFO:tasks.rados.rados.0.smithi105.stdout:1501: seq_num 286 ranges {406802=626275,1686203=728431} 2024-04-20T02:52:18.092 INFO:tasks.rados.rados.0.smithi105.stdout:1501: writing smithi10548823-31 from 406802 to 1033077 tid 1 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1501: writing smithi10548823-31 from 1686203 to 2414634 tid 2 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1484: done (10 left) 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1492: done (9 left) 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1493: done (8 left) 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1494: done (7 left) 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1495: done (6 left) 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1496: done (5 left) 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1497: done (4 left) 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1499: done (3 left) 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1500: done (2 left) 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1502: read oid 39 snap -1 2024-04-20T02:52:18.104 INFO:tasks.rados.rados.0.smithi105.stdout:1502: expect (ObjNum 283 snap 27 seq_num 283) 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1498: finishing write tid 3 to smithi10548823-50 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1498: finishing write tid 4 to smithi10548823-50 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1498: oid 50 updating version 0 to 87 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1498: oid 50 updating version 87 to 88 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1498: oid 50 updating version 88 to 89 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 89 (ObjNum 285 snap 29 seq_num 285) dirty exists 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1498: left oid 50 (ObjNum 285 snap 29 seq_num 285) 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1503: read oid 42 snap -1 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1503: expect (ObjNum 270 snap 26 seq_num 270) 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1498: done (3 left) 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1504: read oid 29 snap -1 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1504: expect (ObjNum 269 snap 25 seq_num 269) 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1505: read oid 10 snap -1 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1505: expect deleted 2024-04-20T02:52:18.105 INFO:tasks.rados.rados.0.smithi105.stdout:1506: read oid 19 snap -1 2024-04-20T02:52:18.106 INFO:tasks.rados.rados.0.smithi105.stdout:1506: expect (ObjNum 278 snap 26 seq_num 278) 2024-04-20T02:52:18.106 INFO:tasks.rados.rados.0.smithi105.stdout:1507: read oid 12 snap -1 2024-04-20T02:52:18.106 INFO:tasks.rados.rados.0.smithi105.stdout:1507: expect deleted 2024-04-20T02:52:18.106 INFO:tasks.rados.rados.0.smithi105.stdout:1508: tier_promote oid 21 2024-04-20T02:52:18.106 INFO:tasks.rados.rados.0.smithi105.stdout:1509: rollback oid 32 current snap is 29 2024-04-20T02:52:18.106 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 32 to 28 2024-04-20T02:52:18.106 INFO:tasks.rados.rados.0.smithi105.stdout:1510: read oid 45 snap -1 2024-04-20T02:52:18.106 INFO:tasks.rados.rados.0.smithi105.stdout:1510: expect (ObjNum 220 snap 21 seq_num 220) 2024-04-20T02:52:18.106 INFO:tasks.rados.rados.0.smithi105.stdout:1511: write (excl) oid 6 current snap is 29 2024-04-20T02:52:18.106 INFO:tasks.rados.rados.0.smithi105.stdout:1511: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-04-20T02:52:18.110 INFO:tasks.rados.rados.0.smithi105.stdout:1511: writing smithi10548823-6 from 423609 to 1125133 tid 1 2024-04-20T02:52:18.116 INFO:tasks.rados.rados.0.smithi105.stdout:1511: writing smithi10548823-6 from 1628332 to 2100421 tid 2 2024-04-20T02:52:18.121 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:18.116+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:18.121 INFO:tasks.rados.rados.0.smithi105.stdout:1511: writing smithi10548823-6 from 2648987 to 2823609 tid 3 2024-04-20T02:52:18.121 INFO:tasks.rados.rados.0.smithi105.stdout:1512: snap_remove snap 22 2024-04-20T02:52:18.148 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:52:18.214 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:18.214+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:18.237 INFO:tasks.rados.rados.0.smithi105.stdout:1512: done (11 left) 2024-04-20T02:52:18.237 INFO:tasks.rados.rados.0.smithi105.stdout:1513: read oid 8 snap 26 2024-04-20T02:52:18.237 INFO:tasks.rados.rados.0.smithi105.stdout:1513: expect (ObjNum 280 snap 26 seq_num 280) 2024-04-20T02:52:18.237 INFO:tasks.rados.rados.0.smithi105.stdout:1514: set_chunk oid 22 target oid 2024-04-20T02:52:18.249 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:18.250 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:18.263 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:18.315 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:18.315+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:18.397 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:18.397 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:18.407 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:18.449 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:18.449+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:18.516 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:18.516+0000 7f34e8e6b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 43300) UID: 0 2024-04-20T02:52:18.575 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:18.575 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:18.586 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:18.587 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:18.627 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:18.627+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:18.718 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:18.719+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:18.760 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:18.760 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:18.771 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:18.775 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:18.794 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:18.812 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:18.819 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:18.820+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:18.831 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:18.850 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:18.868 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:18.887 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:18.904 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:18.905 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:18.905 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:18.905 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:18.906 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:18.907 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:18.907 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:18.907 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:18.908 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:18.908 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:18.908 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:18.908 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:18.908 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:18.916 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:18.960 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:18.960+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:19.021 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:19.022+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:19.085 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:19.086 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:19.086 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:19.086 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:19.086 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:19.095 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:19.136 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:19.137+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:19.222 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:19.223+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:19.244 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:52:19.268 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:19.268 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:19.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:19.268 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:19.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.268 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:19.268 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:19.268 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.323977775000003, 2024-04-20T02:52:19.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:19.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:52:19.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:19.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:52:19.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:52:19.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:52:19.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:52:19.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:52:19.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:52:19.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:52:19.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:52:19.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:52:19.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:52:19.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:52:19.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.316499457000006, 2024-04-20T02:52:19.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:19.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:52:19.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:52:19.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:52:19.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:52:19.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:52:19.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:52:19.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:52:19.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:52:19.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:52:19.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:52:19.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:52:19.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:52:19.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:52:19.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:52:19.284 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.284 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.285 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.285 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:19.285 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:19.285 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.311741557999994, 2024-04-20T02:52:19.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:52:19.285 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.285 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:19.285 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:52:19.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:52:19.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:52:19.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:52:19.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:52:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:52:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:19.291 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.888312623999994, 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:19.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:52:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:52:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:52:19.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:52:19.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:52:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:52:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:52:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:52:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.888275741000001, 2024-04-20T02:52:19.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:52:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:52:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.301 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:19.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:19.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:19.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:52:19.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:52:19.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:52:19.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:19.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:52:19.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:52:19.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:52:19.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:52:19.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:19.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:52:19.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:52:19.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:52:19.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:52:19.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:52:19.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:52:19.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:19.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:52:19.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:52:19.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:52:19.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:52:19.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:52:19.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:52:19.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.313 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:52:19.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:52:19.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:52:19.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:19.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:52:19.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:52:19.315 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.315 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.316 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.316 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:19.316 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:19.316 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.888251108000006, 2024-04-20T02:52:19.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:52:19.317 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.317 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:19.317 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.317 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.318 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.318 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:52:19.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.318 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:52:19.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:52:19.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:52:19.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:52:19.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:52:19.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:52:19.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:52:19.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.323 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.881757696999998, 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:19.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:19.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:52:19.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:52:19.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.331 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.875693674999994, 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.332 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:52:19.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:19.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:52:19.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:52:19.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:19.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:52:19.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:52:19.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:52:19.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:52:19.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:19.339 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:19.340 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.865631910000005, 2024-04-20T02:52:19.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:52:19.340 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.340 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.340 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.340 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.340 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.340 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:52:19.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.340 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:52:19.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:52:19.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:52:19.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:19.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:52:19.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:52:19.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:52:19.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:52:19.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:52:19.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:52:19.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:52:19.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:52:19.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.348 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:19.348 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:19.348 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.864939305999997, 2024-04-20T02:52:19.348 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:52:19.348 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.348 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:19.349 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:52:19.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:52:19.354 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 84.864927652000006, 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.355 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:19.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:52:19.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.360 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.158863096000005, 2024-04-20T02:52:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:52:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.361 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:52:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:19.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:19.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:52:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:52:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:52:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:52:19.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.139802380000006, 2024-04-20T02:52:19.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:52:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:52:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:52:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:52:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:52:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:52:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:52:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:52:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:52:19.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:52:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:52:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:52:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:52:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:52:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:52:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:52:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:52:19.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:52:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.377 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.133856066000007, 2024-04-20T02:52:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:52:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.378 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:52:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:52:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:52:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:52:19.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:52:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:52:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:52:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:52:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:52:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:52:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:52:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:52:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:52:19.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:52:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:52:19.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:52:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:52:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:52:19.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:52:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:52:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:52:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:52:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:52:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:52:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.391 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:19.392 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.133290625000001, 2024-04-20T02:52:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:52:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.393 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:52:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:52:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:52:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:52:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:52:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:52:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:52:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:52:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:52:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:52:19.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:52:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:52:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:52:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:52:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:52:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:52:19.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:52:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:52:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:52:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.403 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.133264593999996, 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:52:19.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:52:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:52:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:52:19.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.131051793000001, 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.410 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:52:19.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:52:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:52:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.415 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.125702269000001, 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:52:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:52:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:52:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:52:19.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:52:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:52:19.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.422 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.125232312000001, 2024-04-20T02:52:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:52:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.423 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:52:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:52:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:52:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:52:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:52:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:52:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:52:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:52:19.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.125209775999998, 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.429 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:52:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:52:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:52:19.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:52:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:52:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:52:19.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:52:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:52:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:52:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:52:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:52:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:52:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:52:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.437 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.438 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:19.441 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:19.441 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:19.440+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:19.532 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:52:19.532 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:52:19.533 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-04-20T02:52:19.533 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:52:19.533 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-20T02:52:19.534 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:52:19.540 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:19.540+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:19.564 INFO:tasks.ceph.osd.2:Stopped 2024-04-20T02:52:19.564 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2 2024-04-20T02:52:19.613 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:19.613 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:19.613 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:19.613 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:19.624 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:19.647 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:19.648+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:19.742 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:19.742+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:19.753 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:19.753 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:19.753 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:19.753 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:19.753 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:19.764 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:19.843 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:19.844+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:19.895 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:19.895 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:19.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:19.895 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:19.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.895 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:19.895 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:19.895 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.159035618999994, 2024-04-20T02:52:19.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:52:19.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:52:19.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:52:19.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:19.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:52:19.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:52:19.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:52:19.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:52:19.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:52:19.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:52:19.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:52:19.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:52:19.904 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.904 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.904 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.904 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:19.904 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:19.904 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.127746557999998, 2024-04-20T02:52:19.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:52:19.904 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.906 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:19.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:52:19.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:52:19.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:52:19.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:52:19.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:52:19.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:52:19.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:52:19.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:52:19.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:19.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:52:19.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:52:19.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:52:19.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:52:19.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:52:19.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:52:19.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:52:19.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:52:19.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.912 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.123056102000007, 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.913 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:19.914 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.916 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:52:19.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:52:19.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.916 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:52:19.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:52:19.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:52:19.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:19.920 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.116299632999997, 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:19.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:52:19.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:52:19.923 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:52:19.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:19.927 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.106484121999998, 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:52:19.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.930 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:52:19.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:52:19.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.933 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.105440189999996, 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:52:19.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:52:19.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.939 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.105419029999993, 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.944 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.897848448999994, 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:52:19.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.948 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:19.949 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.883653804000005, 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:52:19.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.877686546999996, 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:52:19.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.957 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.958 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.869280544000006, 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.959 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.962 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.765648502999994, 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:19.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:52:19.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.535052192999999, 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.967 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:19.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:52:19.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:52:19.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.508108043999997, 2024-04-20T02:52:19.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:52:19.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.974 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.975 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.482749776999995, 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:52:19.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:52:19.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.454609332999993, 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.980 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:19.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:52:19.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:52:19.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:52:19.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:52:19.984 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.417240727999996, 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.985 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:52:19.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:52:19.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:52:19.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:19.989 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.989 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.989 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.416943805000002, 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.411860687000001, 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:52:19.994 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:52:19.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:52:19.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:19.998 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.384858128000005, 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:19.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:52:20.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:52:20.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:52:20.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:52:20.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:52:20.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.003 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.004 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.004 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:20.005 INFO:tasks.ceph.mon.a.smithi105.stderr:2024-04-20T02:52:19.943+0000 7f7f0b14c700 -1 mon.a@0(leader).osd e121 definitely_dead 0 2024-04-20T02:52:20.006 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:20.005+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:20.006 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:20.106 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:20.107+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:20.136 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:20.136 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:20.136 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:20.136 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:20.146 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:20.207 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:20.207+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:20.253 INFO:teuthology.orchestra.run.smithi105.stderr:marked down osd.2. 2024-04-20T02:52:20.266 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 6, 7, 5] 2024-04-20T02:52:20.266 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-04-20T02:52:20.309 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:20.308+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:20.321 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:20.321 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:20.321 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:20.321 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:20.321 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:20.331 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:20.415 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:20.416+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:20.503 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:20.503 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:20.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:20.504 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:20.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.504 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:52:20.504 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:20.504 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 71.054265670999996, 2024-04-20T02:52:20.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:52:20.504 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.504 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.504 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:52:20.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:52:20.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:52:20.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:52:20.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:52:20.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:52:20.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:52:20.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:52:20.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:52:20.508 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.047757438999994, 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034212370999999998, 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456260+0000", 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011140480000000001 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456273+0000", 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456280+0000", 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1130000000000003e-06 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456331+0000", 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1446999999999997e-05 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456357+0000", 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6001999999999999e-05 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489309+0000", 2024-04-20T02:52:20.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032951972000000003 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489359+0000", 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9552000000000003e-05 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1775 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'41, mlcod=53'38)", 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.749375948999997, 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038672274999999999, 2024-04-20T02:52:20.513 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754868+0000", 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013404619999999999 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754881+0000", 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.323e-05 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754892+0000", 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1154e-05 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754967+0000", 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4326999999999995e-05 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.755006+0000", 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9338000000000002e-05 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792154+0000", 2024-04-20T02:52:20.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037147924999999998 2024-04-20T02:52:20.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792200+0000", 2024-04-20T02:52:20.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6044999999999997e-05 2024-04-20T02:52:20.517 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.517 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1822 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'94, mlcod=55'92)", 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.399353816000001, 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031823907999999998, 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:20.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103786+0000", 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000236658 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103797+0000", 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0951000000000001e-05 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103804+0000", 2024-04-20T02:52:20.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0960000000000003e-06 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103860+0000", 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5781e-05 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103887+0000", 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6999999999999999e-05 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135333+0000", 2024-04-20T02:52:20.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031446004 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135374+0000", 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0778999999999997e-05 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1836 4.1 e58/46 4:8b927dd6:::smithi10548823-28:head v 58'99, mlcod=56'96)", 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:52:20.522 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 65.017337807999994, 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030374581000000001, 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485930+0000", 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00036455800000000002 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485941+0000", 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1017e-05 2024-04-20T02:52:20.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485950+0000", 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.85e-06 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486000+0000", 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9589999999999998e-05 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486025+0000", 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.529e-05 2024-04-20T02:52:20.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515909+0000", 2024-04-20T02:52:20.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029884131000000001 2024-04-20T02:52:20.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515941+0000", 2024-04-20T02:52:20.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1569000000000002e-05 2024-04-20T02:52:20.527 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.527 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.527 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.527 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:20.527 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:20.527 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 62.004082904000001, 2024-04-20T02:52:20.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:52:20.528 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.528 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:20.528 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:20.528 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:20.528 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:20.528 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:52:20.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.528 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:52:20.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.532 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 62.004017455000003, 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:20.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 62.003991956999997, 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:20.537 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:20.538 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:20.538 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:52:20.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.538 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.538 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.538 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:52:20.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:52:20.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:52:20.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:52:20.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:52:20.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2004 4.13 e65/63 4:cfd1e4d4:::smithi10548823-9:head v 65'159, mlcod=64'156)", 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.654766909000003, 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036995865000000003, 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849186+0000", 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010492889999999999 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849197+0000", 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1365999999999999e-05 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849204+0000", 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1069999999999999e-06 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849250+0000", 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5213999999999998e-05 2024-04-20T02:52:20.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849279+0000", 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9149000000000002e-05 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885093+0000", 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035813921999999998 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885133+0000", 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0077999999999998e-05 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.547 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2523 4.8 4:1afa5d2b:::smithi10548823-17:head [write 430312~711668,stat] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.279831+0000", 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.223072363999997, 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046423866000000001, 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2523 2024-04-20T02:52:20.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.279831+0000", 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.279831+0000", 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.279830+0000", 2024-04-20T02:52:20.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:20.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280575+0000", 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074521600000000004 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280588+0000", 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.344e-05 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280597+0000", 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3259999999999996e-06 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280639+0000", 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.295e-05 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280755+0000", 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000115988 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280903+0000", 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014755200000000001 2024-04-20T02:52:20.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326112+0000", 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045208950999999997 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326145+0000", 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3235999999999998e-05 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326173+0000", 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8242000000000002e-05 2024-04-20T02:52:20.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326206+0000", 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2398999999999999e-05 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326255+0000", 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9455e-05 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2524 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1723287~549075] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.216482474000003, 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054106970999999997, 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2524 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286419+0000", 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286924+0000", 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050494600000000004 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286944+0000", 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9833000000000001e-05 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286957+0000", 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2901000000000001e-05 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298074+0000", 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011117320999999999 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298151+0000", 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6909999999999994e-05 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298274+0000", 2024-04-20T02:52:20.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012279700000000001 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326267+0000", 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027993284 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331226+0000", 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049590559999999999 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340442+0000", 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092155740000000007 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340488+0000", 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6208000000000002e-05 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340528+0000", 2024-04-20T02:52:20.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0277000000000001e-05 2024-04-20T02:52:20.561 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.561 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.561 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2525 4.8 4:1afa5d2b:::smithi10548823-17:head [write 2781837~703343] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.209731259999998, 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050020689, 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2525 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293172+0000", 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293838+0000", 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066662700000000004 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293857+0000", 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.929e-05 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293867+0000", 2024-04-20T02:52:20.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9369999999999999e-06 2024-04-20T02:52:20.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310149+0000", 2024-04-20T02:52:20.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016281713999999999 2024-04-20T02:52:20.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310257+0000", 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108262 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310439+0000", 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018146300000000001 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.327814+0000", 2024-04-20T02:52:20.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017375043 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331934+0000", 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041200489999999998 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343112+0000", 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011177991999999999 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343146+0000", 2024-04-20T02:52:20.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3813000000000003e-05 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343193+0000", 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2526 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (57) in=7b,truncate 3630312] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.293877+0000", 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.209026686000001, 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049424889, 2024-04-20T02:52:20.569 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2526 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293877+0000", 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293877+0000", 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293876+0000", 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293881+0000", 2024-04-20T02:52:20.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.152e-06 2024-04-20T02:52:20.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293891+0000", 2024-04-20T02:52:20.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5529999999999998e-06 2024-04-20T02:52:20.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293900+0000", 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3449999999999995e-06 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.325730+0000", 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031829673000000003 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.325823+0000", 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2812000000000005e-05 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.325929+0000", 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106263 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.327842+0000", 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019130060000000001 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331975+0000", 2024-04-20T02:52:20.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041330780000000001 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343226+0000", 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011250657000000001 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343260+0000", 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4025e-05 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343302+0000", 2024-04-20T02:52:20.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2199e-05 2024-04-20T02:52:20.576 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.576 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.576 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.576 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2527 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:20.576 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.208998389999998, 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049568319, 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2527 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293908+0000", 2024-04-20T02:52:20.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.605e-06 2024-04-20T02:52:20.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293915+0000", 2024-04-20T02:52:20.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9589999999999998e-06 2024-04-20T02:52:20.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293919+0000", 2024-04-20T02:52:20.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6909999999999996e-06 2024-04-20T02:52:20.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326286+0000", 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032366250999999999 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326340+0000", 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4898000000000002e-05 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343324+0000", 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016983682 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343400+0000", 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5674999999999996e-05 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343474+0000", 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3948000000000004e-05 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2551 4.3 e91/60 4:c07c6daa:::smithi10548823-7:head v 91'204, mlcod=88'201)", 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.453550+0000", 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.049353551999999, 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043524752, 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.453550+0000", 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.453550+0000", 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.453549+0000", 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.454841+0000", 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0012916830000000001 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.454864+0000", 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2690000000000001e-05 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.454871+0000", 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2150000000000004e-06 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.454931+0000", 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0035999999999999e-05 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.454960+0000", 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8873999999999998e-05 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.497040+0000", 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042079977999999997 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.497075+0000", 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2733 4.1 e95/60 4:80c7a726:::smithi10548823-19:head v 95'104, mlcod=58'101)", 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:52.432024+0000", 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.070879316999999, 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045945311000000003, 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432024+0000", 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432024+0000", 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432024+0000", 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432220+0000", 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000196054 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432231+0000", 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1044e-05 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432236+0000", 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1590000000000001e-06 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432279+0000", 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3541000000000002e-05 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432304+0000", 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4399000000000001e-05 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.477933+0000", 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045628902999999998 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.477970+0000", 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7017999999999997e-05 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2832 3.2 e97/49 3:487ee27a:::27b240b05ce75114b683ef257328c902fdd5cc3998dce9a6304f8777ca278278:head v 97'162, mlcod=97'160)", 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.080828+0000", 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.422075735, 2024-04-20T02:52:20.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041490463999999998, 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.080828+0000", 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.080828+0000", 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.080827+0000", 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.084247+0000", 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034197089999999999 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.084266+0000", 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9706000000000001e-05 2024-04-20T02:52:20.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.084282+0000", 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.607e-05 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.084351+0000", 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8918000000000007e-05 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.084383+0000", 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1458999999999998e-05 2024-04-20T02:52:20.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.122274+0000", 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037890857999999999 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.122319+0000", 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4996000000000001e-05 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2899 3.2 e103/49 3:79b354f1:::65c4a6cbe46e8d949445597300a5482d595a2e84e6ac0aa957f292131d610636:head v 103'167, mlcod=101'165)", 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:01.145855+0000", 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.357048421000002, 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040012859999999997, 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.145855+0000", 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.145855+0000", 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.145854+0000", 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.147738+0000", 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0018831690000000001 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.147748+0000", 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0363000000000001e-05 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.147754+0000", 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0510000000000002e-06 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.147796+0000", 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2159000000000001e-05 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.147814+0000", 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7365999999999999e-05 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.185831+0000", 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038017185000000002 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.185868+0000", 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7460999999999999e-05 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.599 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2916 4.18 4:1afa5d2b:::smithi10548823-17:head [write 635364~402397,stat] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:01.621072+0000", 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.881832201000002, 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032500330000000001, 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2916 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621072+0000", 2024-04-20T02:52:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621072+0000", 2024-04-20T02:52:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621069+0000", 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621522+0000", 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00045231200000000002 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621543+0000", 2024-04-20T02:52:20.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1282000000000001e-05 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621555+0000", 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2218999999999999e-05 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621604+0000", 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8420000000000001e-05 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621684+0000", 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0706999999999998e-05 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621838+0000", 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000153845 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.647648+0000", 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025809622000000001 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.648284+0000", 2024-04-20T02:52:20.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063657000000000004 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.653450+0000", 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0051652110000000003 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.653508+0000", 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8504000000000001e-05 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.653572+0000", 2024-04-20T02:52:20.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3788999999999995e-05 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2936 3.2 e104/49 3:4b83f5f2:::b2faa4f12d7cc0705e9572ec0b15996379add8e079fd680499ccc7725ce92280:head v 104'172, mlcod=103'170)", 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:02.186116+0000", 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.316787294000001, 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033070583000000001, 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:20.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.186116+0000", 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.186116+0000", 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.186116+0000", 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:20.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.187683+0000", 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001567316 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.187695+0000", 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2279000000000001e-05 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.187704+0000", 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3839999999999997e-06 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.187753+0000", 2024-04-20T02:52:20.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9706000000000001e-05 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.187784+0000", 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1165999999999999e-05 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.219135+0000", 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031350590999999997 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.219187+0000", 2024-04-20T02:52:20.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2023e-05 2024-04-20T02:52:20.610 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.610 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.610 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.610 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:20.610 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:20.610 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:20.611 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:20.612+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:20.612 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:20.650 INFO:tasks.ceph.mon.a.smithi105.stderr:2024-04-20T02:52:20.650+0000 7f7f0b14c700 -1 mon.a@0(leader).osd e122 definitely_dead 0 2024-04-20T02:52:20.712 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:20.712+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:20.740 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:20.740 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:20.741 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:20.741 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:20.750 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:20.813 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:20.814+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:20.913 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:20.914+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:20.918 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:20.922 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:20.922 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:20.923 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:20.923 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:20.923 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:20.929 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:21.014 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:21.015+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:21.070 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:21.091 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:21.104 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:21.105 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:21.110 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:21.115 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:21.116 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:21.116+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:21.133 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:21.153 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:21.173 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:21.192 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:21.193 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:21.193 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:21.193 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T02:52:21.193 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:21.193 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:21.193 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:21.193 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:21.193 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:21.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:21.238 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:21.238+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:21.248 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:21.249 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:21.255 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:52:21.258 INFO:teuthology.orchestra.run.smithi105.stderr:marked out osd.2. 2024-04-20T02:52:21.265 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:21.272 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-04-20T02:52:21.273 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:52:21.273 DEBUG:teuthology.orchestra.run.smithi105:> type ceph-objectstore-tool 2024-04-20T02:52:21.328 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-20T02:52:21.329 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-04-20T02:52:21.329 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:21.328+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:21.417 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:21.418+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:21.459 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:21.459 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:21.471 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:21.518 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:21.519+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:21.619 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:21.620+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:21.640 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:21.640 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:21.651 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:21.720 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:21.720+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:21.786 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:21.786 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:21.801 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:21.821 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:21.821+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:21.921 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:21.922+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:21.932 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:21.932 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:21.942 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:22.022 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:22.023+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:22.073 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:22.073 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:22.084 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:22.129 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:22.130+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:22.225 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:22.226+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:22.227 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:22.227 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:22.237 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:22.264 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:52:22.326 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:22.327+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:22.406 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:22.406 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:22.416 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:22.458 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:22.458+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:22.504 INFO:teuthology.orchestra.run.smithi105.stdout:4.d 2024-04-20T02:52:22.504 INFO:teuthology.orchestra.run.smithi105.stdout:4.4 2024-04-20T02:52:22.504 INFO:teuthology.orchestra.run.smithi105.stdout:4.6 2024-04-20T02:52:22.504 INFO:teuthology.orchestra.run.smithi105.stdout:4.16 2024-04-20T02:52:22.504 INFO:teuthology.orchestra.run.smithi105.stdout:4.14 2024-04-20T02:52:22.504 INFO:teuthology.orchestra.run.smithi105.stdout:3.0 2024-04-20T02:52:22.504 INFO:teuthology.orchestra.run.smithi105.stdout:2.7 2024-04-20T02:52:22.504 INFO:teuthology.orchestra.run.smithi105.stdout:2.3 2024-04-20T02:52:22.504 INFO:teuthology.orchestra.run.smithi105.stdout:2.1 2024-04-20T02:52:22.504 INFO:teuthology.orchestra.run.smithi105.stdout:2.0 2024-04-20T02:52:22.528 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:22.528+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:22.586 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:22.586 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:22.596 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:22.639 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:22.640+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:22.725 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:22.725 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:22.729 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:22.729+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:22.735 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:22.830 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:22.831+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:22.905 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:22.905 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:22.915 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:22.932 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:22.932+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:23.021 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 4.16 --file /var/log/ceph/exp.4.16.2 2024-04-20T02:52:23.033 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:23.033+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:23.084 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:23.084 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:23.094 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:23.135 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:23.136+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:23.197 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:23.235 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:23.236+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:23.270 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:23.271 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:23.281 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:23.289 INFO:tasks.rados.rados.0.smithi105.stdout:1514: set_chunk oid 22 offset: 668160 length: 797184 target oid 74e23170e508eda89cb4f2d8f91f2e9f6d0895d458e1003cd0f4ba60d2ea9ba6 offset: 0 snap -1 2024-04-20T02:52:23.289 INFO:tasks.rados.rados.0.smithi105.stdout:1515: read oid 33 snap -1 2024-04-20T02:52:23.289 INFO:tasks.rados.rados.0.smithi105.stdout:1515: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:52:23.289 INFO:tasks.rados.rados.0.smithi105.stdout:1516: set_chunk oid 16 target oid 2024-04-20T02:52:23.289 INFO:tasks.rados.rados.0.smithi105.stdout:1516: set_chunk oid 16 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:23.289 INFO:tasks.rados.rados.0.smithi105.stdout:1517: read oid 14 snap -1 2024-04-20T02:52:23.290 INFO:tasks.rados.rados.0.smithi105.stdout:1517: expect (ObjNum 274 snap 26 seq_num 274) 2024-04-20T02:52:23.290 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:23.290 INFO:tasks.rados.rados.0.smithi105.stdout:1505: done (15 left) 2024-04-20T02:52:23.290 INFO:tasks.rados.rados.0.smithi105.stdout:1507: done (14 left) 2024-04-20T02:52:23.290 INFO:tasks.rados.rados.0.smithi105.stdout:1518: set_chunk oid 49 target oid 2024-04-20T02:52:23.323 INFO:tasks.rados.rados.0.smithi105.stdout:1518: set_chunk oid 49 offset: 1220608 length: 1000960 target oid 5557163f5a34185a694ec28243952dd5855ee132930328131757db35ae4dbff6 offset: 0 snap 26 2024-04-20T02:52:23.323 INFO:tasks.rados.rados.0.smithi105.stdout:1519: snap_create 2024-04-20T02:52:23.324 INFO:tasks.rados.rados.0.smithi105.stdout:1502: expect (ObjNum 283 snap 27 seq_num 283) 2024-04-20T02:52:23.337 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:23.337+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:23.377 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:23.402 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:23.422 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:23.437 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:23.438+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:23.444 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:23.467 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:23.468 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:23.468 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:23.480 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:23.490 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:23.510 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:23.511 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:23.511 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:23.511 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T02:52:23.511 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:23.511 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:23.511 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:23.511 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:23.511 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:23.513 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:23.514 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:23.524 INFO:tasks.rados.rados.0.smithi105.stdout:1508: got (0) Success 2024-04-20T02:52:23.524 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 113 (ObjNum 59212080 snap 22058 seq_num 304) clean dne 2024-04-20T02:52:23.524 INFO:tasks.rados.rados.0.smithi105.stdout:1504: expect (ObjNum 269 snap 25 seq_num 269) 2024-04-20T02:52:23.538 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:23.538+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:23.639 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:23.639+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 124.04642004199999, 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:23.651 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:52:23.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:23.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:52:23.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:52:23.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:52:23.658 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 102.282553643, 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:52:23.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:52:23.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 102.16925060200001, 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:23.663 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:52:23.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:52:23.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:52:23.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:52:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:52:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:23.669 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 102.163485085, 2024-04-20T02:52:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:52:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.674 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 101.80946495000001, 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:52:23.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:52:23.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:52:23.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 101.799101345, 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:52:23.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 101.79101686600001, 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 83.509458640999995, 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.688 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:52:23.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:52:23.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:52:23.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 83.504321531000002, 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:52:23.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:52:23.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 83.504048518999994, 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:23.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:23.705 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 83.504019134000004, 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:52:23.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:52:23.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:23.710 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 73.195125168999994, 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035060916999999997, 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456603+0000", 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001394295 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456616+0000", 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2605999999999999e-05 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456631+0000", 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4735e-05 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456682+0000", 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1048000000000003e-05 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456705+0000", 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2881e-05 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490235+0000", 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033530250999999997 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490270+0000", 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5379000000000001e-05 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 60.816691835, 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:23.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:52:23.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.718 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2132 4.10 e70/63 4:0b0a7da5:::smithi10548823-43:head v 70'76, mlcod=65'73)", 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 55.906093796, 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036823724000000002, 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:23.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744239+0000", 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744631+0000", 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039175199999999999 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744644+0000", 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3079e-05 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744653+0000", 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8629999999999997e-06 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744706+0000", 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2627999999999998e-05 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744737+0000", 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1387999999999998e-05 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781039+0000", 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036302147 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781064+0000", 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5182999999999999e-05 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2293 4.3 e76/60 4:c07c6daa:::smithi10548823-7:head v 76'165, mlcod=64'161)", 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 49.461225773000002, 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034106808000000002, 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.723 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189108+0000", 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:23.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190758+0000", 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016504740000000001 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190778+0000", 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9507000000000001e-05 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190787+0000", 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1740000000000006e-06 2024-04-20T02:52:23.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190836+0000", 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9351999999999998e-05 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190854+0000", 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7629e-05 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.223165+0000", 2024-04-20T02:52:23.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032310885999999997 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.223216+0000", 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0615000000000002e-05 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.369211612999997, 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:52:23.727 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:52:23.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:52:23.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.351943179000003, 2024-04-20T02:52:23.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.14 98/97 [PushOp(4:2e88ea27:::smithi10548823-12:head, version: 98'33, data_included: [406769~907695,1439434~140881], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:2e88ea27:::smithi10548823-12:head@98'33, size: 2033680, copy_subset: [0~2033680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1580315, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:23.735 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.400176344999998, 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033699081999999998, 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251322+0000", 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001164251 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251340+0000", 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.789e-05 2024-04-20T02:52:23.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251356+0000", 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5962000000000001e-05 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251428+0000", 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2750000000000007e-05 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251444+0000", 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5248e-05 2024-04-20T02:52:23.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.283857+0000", 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032413659999999997 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.6.0:82 4.15 e104/103 4:a9897a5e:::smithi10548823-42:head v 104'64, mlcod=93'62)", 2024-04-20T02:52:23.739 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 21.514462050999999, 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035573920000000002, 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:23.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135871+0000", 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136395+0000", 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00052370100000000001 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136409+0000", 2024-04-20T02:52:23.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4191e-05 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136418+0000", 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0100000000000001e-06 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136468+0000", 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9458e-05 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136499+0000", 2024-04-20T02:52:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1062999999999997e-05 2024-04-20T02:52:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.171417+0000", 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034918607999999997 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.171446+0000", 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9000999999999999e-05 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:52:23.744 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.178797383999999, 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:23.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:23.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:23.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:23.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:23.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:52:23.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:52:23.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:23.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.749 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.750 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:23.750 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:23.750 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:23.751 INFO:tasks.rados.rados.0.smithi105.stdout:1506: expect (ObjNum 278 snap 26 seq_num 278) 2024-04-20T02:52:23.752 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:23.752 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:23.751+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:23.852 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:23.852+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:23.877 INFO:tasks.rados.rados.0.smithi105.stdout:1503: expect (ObjNum 270 snap 26 seq_num 270) 2024-04-20T02:52:23.922 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:23.922 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:23.922 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:23.922 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:23.932 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:23.953 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:23.953+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:24.019 INFO:tasks.rados.rados.0.smithi105.stdout:1510: expect (ObjNum 220 snap 21 seq_num 220) 2024-04-20T02:52:24.053 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:24.054+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:24.060 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:24.060 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:24.060 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:24.060 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:24.060 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:24.071 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:24.149 INFO:teuthology.orchestra.run.smithi105.stderr:Exporting 4.16 info 4.16( v 117'91 (101'85,117'91] local-lis/les=120/121 n=3 ec=60/22 lis/c=120/120 les/c/f=121/121/0 sis=120) 2024-04-20T02:52:24.154 INFO:teuthology.orchestra.run.smithi105.stderr:Read #4:68acf93d:::smithi10548823-22:32# 2024-04-20T02:52:24.154 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:24.154+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:24.157 INFO:teuthology.orchestra.run.smithi105.stderr:Read #4:68acf93d:::smithi10548823-22:head# 2024-04-20T02:52:24.160 INFO:teuthology.orchestra.run.smithi105.stderr:Read #4:697da67d:::smithi10548823-48:head# 2024-04-20T02:52:24.166 INFO:teuthology.orchestra.run.smithi105.stderr:Export successful 2024-04-20T02:52:24.193 INFO:tasks.rados.rados.0.smithi105.stdout:1509: finishing rollback tid 1 to smithi10548823-32 2024-04-20T02:52:24.193 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 0 (ObjNum 257 snap 0 seq_num 59080320) dirty dne 2024-04-20T02:52:24.193 INFO:tasks.rados.rados.0.smithi105.stdout:1501: finishing write tid 1 to smithi10548823-31 2024-04-20T02:52:24.193 INFO:tasks.rados.rados.0.smithi105.stdout:1501: finishing write tid 2 to smithi10548823-31 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1501: finishing write tid 3 to smithi10548823-31 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1501: finishing write tid 4 to smithi10548823-31 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1501: oid 31 updating version 0 to 185 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1501: oid 31 updating version 185 to 187 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1501: oid 31 version 187 is already newer than 186 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 187 (ObjNum 286 snap 29 seq_num 286) dirty exists 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1501: left oid 31 (ObjNum 286 snap 29 seq_num 286) 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1511: finishing write tid 1 to smithi10548823-6 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1511: finishing write tid 2 to smithi10548823-6 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1511: finishing write tid 3 to smithi10548823-6 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1511: finishing write tid 4 to smithi10548823-6 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1511: finishing write tid 5 to smithi10548823-6 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1511: oid 6 updating version 0 to 176 2024-04-20T02:52:24.194 INFO:tasks.rados.rados.0.smithi105.stdout:1511: oid 6 version 176 is already newer than 175 2024-04-20T02:52:24.195 INFO:tasks.rados.rados.0.smithi105.stdout:1511: oid 6 updating version 176 to 177 2024-04-20T02:52:24.195 INFO:tasks.rados.rados.0.smithi105.stdout:1511: oid 6 updating version 177 to 178 2024-04-20T02:52:24.195 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 178 (ObjNum 287 snap 29 seq_num 287) dirty exists 2024-04-20T02:52:24.195 INFO:tasks.rados.rados.0.smithi105.stdout:1511: left oid 6 (ObjNum 287 snap 29 seq_num 287) 2024-04-20T02:52:24.195 INFO:tasks.rados.rados.0.smithi105.stdout:1513: expect (ObjNum 280 snap 26 seq_num 280) 2024-04-20T02:52:24.241 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:24.241 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:24.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:24.242 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:24.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.242 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:52:24.242 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:24.242 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 74.792393648000001, 2024-04-20T02:52:24.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:52:24.242 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.242 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.242 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 73.785885415999999, 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034212370999999998, 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.246 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455146+0000", 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456260+0000", 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0011140480000000001 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456273+0000", 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:52:24.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456280+0000", 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1130000000000003e-06 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456331+0000", 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1446999999999997e-05 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456357+0000", 2024-04-20T02:52:24.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6001999999999999e-05 2024-04-20T02:52:24.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489309+0000", 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032951972000000003 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.489359+0000", 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9552000000000003e-05 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1775 4.7 e56/46 4:e2c0d65b:::smithi10548823-46:head v 56'41, mlcod=53'38)", 2024-04-20T02:52:24.251 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.487503926000002, 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038672274999999999, 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.753528+0000", 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967296 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754868+0000", 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0013404619999999999 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754881+0000", 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.323e-05 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754892+0000", 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1154e-05 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.754967+0000", 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4326999999999995e-05 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.755006+0000", 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9338000000000002e-05 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792154+0000", 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037147924999999998 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:13.792200+0000", 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6044999999999997e-05 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1822 4.1 e56/46 4:8efa5349:::smithi10548823-10:head v 56'94, mlcod=55'92)", 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 70.137481793000006, 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031823907999999998, 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103550+0000", 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103786+0000", 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000236658 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103797+0000", 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0951000000000001e-05 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103804+0000", 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.0960000000000003e-06 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103860+0000", 2024-04-20T02:52:24.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5781e-05 2024-04-20T02:52:24.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.103887+0000", 2024-04-20T02:52:24.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6999999999999999e-05 2024-04-20T02:52:24.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135333+0000", 2024-04-20T02:52:24.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031446004 2024-04-20T02:52:24.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:14.135374+0000", 2024-04-20T02:52:24.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0778999999999997e-05 2024-04-20T02:52:24.260 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.260 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.260 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1836 4.1 e58/46 4:8b927dd6:::smithi10548823-28:head v 58'99, mlcod=56'96)", 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 68.755465784999998, 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030374581000000001, 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485566+0000", 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485930+0000", 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00036455800000000002 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485941+0000", 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1017e-05 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.485950+0000", 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.85e-06 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486000+0000", 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9589999999999998e-05 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.486025+0000", 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.529e-05 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515909+0000", 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029884131000000001 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:15.515941+0000", 2024-04-20T02:52:24.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1569000000000002e-05 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 65.742210881000005, 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.265 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:52:24.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:52:24.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:24.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:52:24.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:52:24.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:24.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:52:24.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:52:24.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:52:24.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 65.742145432000001, 2024-04-20T02:52:24.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:52:24.271 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.271 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:24.271 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:24.271 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:24.271 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.271 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:52:24.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.271 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:24.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:52:24.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:52:24.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:52:24.275 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.275 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.275 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.276 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:24.276 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:24.276 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 65.742119934000002, 2024-04-20T02:52:24.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:52:24.276 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.276 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:24.276 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:24.276 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:24.276 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.277 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:52:24.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.277 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:24.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:24.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:52:24.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:52:24.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:52:24.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:52:24.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:52:24.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:52:24.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:52:24.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:52:24.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:52:24.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:24.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:52:24.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:52:24.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:52:24.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:52:24.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:52:24.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:52:24.282 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.282 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.283 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.283 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2004 4.13 e65/63 4:cfd1e4d4:::smithi10548823-9:head v 65'159, mlcod=64'156)", 2024-04-20T02:52:24.283 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:52:24.283 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.392894886000001, 2024-04-20T02:52:24.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036995865000000003, 2024-04-20T02:52:24.283 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.283 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.284 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:52:24.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:52:24.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.848137+0000", 2024-04-20T02:52:24.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849186+0000", 2024-04-20T02:52:24.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010492889999999999 2024-04-20T02:52:24.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849197+0000", 2024-04-20T02:52:24.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1365999999999999e-05 2024-04-20T02:52:24.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849204+0000", 2024-04-20T02:52:24.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1069999999999999e-06 2024-04-20T02:52:24.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849250+0000", 2024-04-20T02:52:24.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5213999999999998e-05 2024-04-20T02:52:24.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.849279+0000", 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9149000000000002e-05 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885093+0000", 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035813921999999998 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.885133+0000", 2024-04-20T02:52:24.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0077999999999998e-05 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2523 4.8 4:1afa5d2b:::smithi10548823-17:head [write 430312~711668,stat] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.279831+0000", 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.961200341000001, 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046423866000000001, 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.289 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2523 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.279831+0000", 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.279831+0000", 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.279830+0000", 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280575+0000", 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074521600000000004 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280588+0000", 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.344e-05 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280597+0000", 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3259999999999996e-06 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280639+0000", 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.295e-05 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280755+0000", 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000115988 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:24.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.280903+0000", 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014755200000000001 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326112+0000", 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045208950999999997 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326145+0000", 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3235999999999998e-05 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326173+0000", 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8242000000000002e-05 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326206+0000", 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2398999999999999e-05 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326255+0000", 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9455e-05 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2524 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1723287~549075] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.954610451000001, 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054106970999999997, 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2524 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286419+0000", 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286924+0000", 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050494600000000004 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286944+0000", 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9833000000000001e-05 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286957+0000", 2024-04-20T02:52:24.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2901000000000001e-05 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298074+0000", 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011117320999999999 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298151+0000", 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6909999999999994e-05 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298274+0000", 2024-04-20T02:52:24.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012279700000000001 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326267+0000", 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027993284 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331226+0000", 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049590559999999999 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340442+0000", 2024-04-20T02:52:24.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092155740000000007 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340488+0000", 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6208000000000002e-05 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340528+0000", 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0277000000000001e-05 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2525 4.8 4:1afa5d2b:::smithi10548823-17:head [write 2781837~703343] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.947859237000003, 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050020689, 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2525 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293172+0000", 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293838+0000", 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066662700000000004 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293857+0000", 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.929e-05 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293867+0000", 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9369999999999999e-06 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310149+0000", 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016281713999999999 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310257+0000", 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108262 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310439+0000", 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018146300000000001 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:24.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.327814+0000", 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017375043 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331934+0000", 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041200489999999998 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343112+0000", 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011177991999999999 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343146+0000", 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3813000000000003e-05 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343193+0000", 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2526 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (57) in=7b,truncate 3630312] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.293877+0000", 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.947154662999999, 2024-04-20T02:52:24.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049424889, 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2526 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293877+0000", 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293877+0000", 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293876+0000", 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293881+0000", 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.152e-06 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293891+0000", 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5529999999999998e-06 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293900+0000", 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3449999999999995e-06 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.325730+0000", 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031829673000000003 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.325823+0000", 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2812000000000005e-05 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.325929+0000", 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106263 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.327842+0000", 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019130060000000001 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331975+0000", 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041330780000000001 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343226+0000", 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011250657000000001 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343260+0000", 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4025e-05 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343302+0000", 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2199e-05 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2527 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.947126367000003, 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049568319, 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:24.312 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2527 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293908+0000", 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.605e-06 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293915+0000", 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9589999999999998e-06 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293919+0000", 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6909999999999996e-06 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326286+0000", 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032366250999999999 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326340+0000", 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4898000000000002e-05 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343324+0000", 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016983682 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343400+0000", 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5674999999999996e-05 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343474+0000", 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3948000000000004e-05 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2551 4.3 e91/60 4:c07c6daa:::smithi10548823-7:head v 91'204, mlcod=88'201)", 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.453550+0000", 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.787481528999997, 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043524752, 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.453550+0000", 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.453550+0000", 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.453549+0000", 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.454841+0000", 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0012916830000000001 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.454864+0000", 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2690000000000001e-05 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.454871+0000", 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2150000000000004e-06 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.454931+0000", 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0035999999999999e-05 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.454960+0000", 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8873999999999998e-05 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.497040+0000", 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042079977999999997 2024-04-20T02:52:24.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.497075+0000", 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2733 4.1 e95/60 4:80c7a726:::smithi10548823-19:head v 95'104, mlcod=58'101)", 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:52.432024+0000", 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 31.809007294000001, 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045945311000000003, 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.321 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432024+0000", 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432024+0000", 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432024+0000", 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432220+0000", 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000196054 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432231+0000", 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1044e-05 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432236+0000", 2024-04-20T02:52:24.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1590000000000001e-06 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432279+0000", 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3541000000000002e-05 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.432304+0000", 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4399000000000001e-05 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.477933+0000", 2024-04-20T02:52:24.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045628902999999998 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:52.477970+0000", 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7017999999999997e-05 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2832 3.2 e97/49 3:487ee27a:::27b240b05ce75114b683ef257328c902fdd5cc3998dce9a6304f8777ca278278:head v 97'162, mlcod=97'160)", 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.080828+0000", 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.160203712000001, 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041490463999999998, 2024-04-20T02:52:24.325 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.080828+0000", 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.080828+0000", 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.080827+0000", 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.084247+0000", 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034197089999999999 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.084266+0000", 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9706000000000001e-05 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.084282+0000", 2024-04-20T02:52:24.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.607e-05 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.084351+0000", 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8918000000000007e-05 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.084383+0000", 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1458999999999998e-05 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.122274+0000", 2024-04-20T02:52:24.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037890857999999999 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.122319+0000", 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4996000000000001e-05 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2899 3.2 e103/49 3:79b354f1:::65c4a6cbe46e8d949445597300a5482d595a2e84e6ac0aa957f292131d610636:head v 103'167, mlcod=101'165)", 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:01.145855+0000", 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.095176398, 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040012859999999997, 2024-04-20T02:52:24.329 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.145855+0000", 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.145855+0000", 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.145854+0000", 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.147738+0000", 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0018831690000000001 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.147748+0000", 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0363000000000001e-05 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.147754+0000", 2024-04-20T02:52:24.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0510000000000002e-06 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.147796+0000", 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2159000000000001e-05 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.147814+0000", 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7365999999999999e-05 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.185831+0000", 2024-04-20T02:52:24.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038017185000000002 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.185868+0000", 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.7460999999999999e-05 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2916 4.18 4:1afa5d2b:::smithi10548823-17:head [write 635364~402397,stat] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:01.621072+0000", 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.619960177999999, 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032500330000000001, 2024-04-20T02:52:24.333 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2916 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621072+0000", 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621072+0000", 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621069+0000", 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621522+0000", 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00045231200000000002 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621543+0000", 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1282000000000001e-05 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621555+0000", 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2218999999999999e-05 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621604+0000", 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8420000000000001e-05 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621684+0000", 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0706999999999998e-05 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.621838+0000", 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000153845 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.647648+0000", 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025809622000000001 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:24.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.648284+0000", 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063657000000000004 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.653450+0000", 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0051652110000000003 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.653508+0000", 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8504000000000001e-05 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:01.653572+0000", 2024-04-20T02:52:24.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3788999999999995e-05 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2936 3.2 e104/49 3:4b83f5f2:::b2faa4f12d7cc0705e9572ec0b15996379add8e079fd680499ccc7725ce92280:head v 104'172, mlcod=103'170)", 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:02.186116+0000", 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.054915270999999, 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033070583000000001, 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.186116+0000", 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.186116+0000", 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.186116+0000", 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:24.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.187683+0000", 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001567316 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.187695+0000", 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2279000000000001e-05 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.187704+0000", 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3839999999999997e-06 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:24.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.187753+0000", 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9706000000000001e-05 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.187784+0000", 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1165999999999999e-05 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.219135+0000", 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031350590999999997 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:24.342 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.219187+0000", 2024-04-20T02:52:24.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2023e-05 2024-04-20T02:52:24.343 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.343 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.343 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.343 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:24.343 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:24.343 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:24.343 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:52:24.345 INFO:tasks.rados.rados.0.smithi105.stdout:1516: finishing set_chunk to oid 16 2024-04-20T02:52:24.345 INFO:tasks.rados.rados.0.smithi105.stdout:1516: got expected ENOENT (src dne) 2024-04-20T02:52:24.345 INFO:tasks.rados.rados.0.smithi105.stdout:1501: done (15 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1502: done (14 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1503: done (13 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1504: done (12 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1506: done (11 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1508: done (10 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1509: done (9 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1510: done (8 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1511: done (7 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1513: done (6 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1516: done (5 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1519: done (4 left) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1520: read oid 44 snap -1 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1520: expect (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1521: read oid 16 snap -1 2024-04-20T02:52:24.346 INFO:tasks.rados.rados.0.smithi105.stdout:1521: expect deleted 2024-04-20T02:52:24.347 INFO:tasks.rados.rados.0.smithi105.stdout:1522: delete oid 42 current snap is 30 2024-04-20T02:52:24.347 INFO:tasks.rados.rados.0.smithi105.stdout:1517: expect (ObjNum 274 snap 26 seq_num 274) 2024-04-20T02:52:24.347 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:24.347 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:24.344+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:24.445 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:24.445+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:24.487 INFO:tasks.rados.rados.0.smithi105.stdout:1515: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:52:24.488 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:24.488 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:24.488 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:24.488 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:24.499 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:24.584 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:24.584+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:24.646 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:24.647+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:24.669 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:24.669 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:24.669 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:24.669 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:24.669 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:24.671 INFO:tasks.rados.rados.0.smithi105.stdout:1514: finishing set_chunk to oid 22 2024-04-20T02:52:24.671 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 89 (ObjNum 273 snap 26 seq_num 273) dirty exists 2024-04-20T02:52:24.671 INFO:tasks.rados.rados.0.smithi105.stdout:1518: finishing set_chunk to oid 49 2024-04-20T02:52:24.671 INFO:tasks.rados.rados.0.smithi105.stdout:1520: expect (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:52:24.679 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:24.689 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 4.16 2024-04-20T02:52:24.747 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:24.748+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:24.848 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:24.848+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:927 4.3 4:cf11ccc0:::smithi10548823-4:head [write 3121996~104915] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.484468480000004, 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058986447999999997, 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.863 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 927 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377548+0000", 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377607+0000", 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9413999999999997e-05 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377616+0000", 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4880000000000007e-06 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377621+0000", 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.682e-06 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407778+0000", 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030156110999999999 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407866+0000", 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8645000000000003e-05 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407985+0000", 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118874 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411741+0000", 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037563060000000001 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411854+0000", 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112898 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436438+0000", 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024583885 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436483+0000", 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5334999999999999e-05 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436535+0000", 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1780000000000002e-05 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.868 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:928 4.3 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (55) in=7b,truncate 3226911] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.484389874000001, 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059064512999999999, 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 928 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377630+0000", 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7259999999999998e-06 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377633+0000", 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1489999999999998e-06 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377636+0000", 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7760000000000002e-06 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410754+0000", 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033117944000000003 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410850+0000", 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6522999999999997e-05 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411008+0000", 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158157 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436151+0000", 2024-04-20T02:52:24.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025142813 2024-04-20T02:52:24.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436160+0000", 2024-04-20T02:52:24.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1460000000000002e-06 2024-04-20T02:52:24.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436588+0000", 2024-04-20T02:52:24.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00042776499999999998 2024-04-20T02:52:24.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436631+0000", 2024-04-20T02:52:24.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3084000000000003e-05 2024-04-20T02:52:24.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436692+0000", 2024-04-20T02:52:24.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0829000000000003e-05 2024-04-20T02:52:24.876 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.876 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.876 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.877 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:929 4.3 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:24.877 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:52:24.877 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.484378733, 2024-04-20T02:52:24.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059214058, 2024-04-20T02:52:24.877 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 929 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377638+0000", 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377640+0000", 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5430000000000001e-06 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377643+0000", 2024-04-20T02:52:24.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0900000000000001e-06 2024-04-20T02:52:24.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377647+0000", 2024-04-20T02:52:24.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2230000000000001e-06 2024-04-20T02:52:24.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411271+0000", 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033623635999999998 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411329+0000", 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8043e-05 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436728+0000", 2024-04-20T02:52:24.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025399300999999999 2024-04-20T02:52:24.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436793+0000", 2024-04-20T02:52:24.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5041000000000003e-05 2024-04-20T02:52:24.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436853+0000", 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9446e-05 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:936 4.3 4:ca036f62:::smithi10548823-34:head [write 443718~615603] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.467343400000004, 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060880228000000002, 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.883 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 936 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394672+0000", 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394825+0000", 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015343299999999999 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394844+0000", 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8414999999999999e-05 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394859+0000", 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4738e-05 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411879+0000", 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017020469999999999 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411954+0000", 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5239000000000002e-05 2024-04-20T02:52:24.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:24.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412104+0000", 2024-04-20T02:52:24.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014944399999999999 2024-04-20T02:52:24.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436402+0000", 2024-04-20T02:52:24.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024297828 2024-04-20T02:52:24.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436409+0000", 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1779999999999997e-06 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455390+0000", 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018981221 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455508+0000", 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117556 2024-04-20T02:52:24.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455554+0000", 2024-04-20T02:52:24.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6570000000000003e-05 2024-04-20T02:52:24.890 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:937 4.3 4:ca036f62:::smithi10548823-34:head [write 1564138~470341] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.462005070000004, 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060119655000000001, 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.891 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 937 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400010+0000", 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400222+0000", 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000211208 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400238+0000", 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419e-05 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400250+0000", 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2091e-05 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425851+0000", 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025600795999999999 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425965+0000", 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011411799999999999 2024-04-20T02:52:24.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426134+0000", 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168517 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436937+0000", 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01080328 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455351+0000", 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018414607 2024-04-20T02:52:24.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460001+0000", 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046491809999999996 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460086+0000", 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5066000000000002e-05 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460132+0000", 2024-04-20T02:52:24.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6174999999999998e-05 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:938 4.3 4:ca036f62:::smithi10548823-34:head [write 2633066~610652] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.455239614000007, 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054360654000000001, 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.897 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 938 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406775+0000", 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406947+0000", 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171619 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406962+0000", 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5248999999999999e-05 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406979+0000", 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7334999999999999e-05 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436954+0000", 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029974218 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437007+0000", 2024-04-20T02:52:24.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3553e-05 2024-04-20T02:52:24.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:24.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437110+0000", 2024-04-20T02:52:24.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:24.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455605+0000", 2024-04-20T02:52:24.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018494295000000001 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455829+0000", 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022443100000000001 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461044+0000", 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052150499999999997 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461090+0000", 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6100999999999998e-05 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461138+0000", 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8003000000000001e-05 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.904 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:939 4.3 4:ca036f62:::smithi10548823-34:head [setxattr _header (56) in=7b,truncate 3243718] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:24.904 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:52:24.904 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.455027302999994, 2024-04-20T02:52:24.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054380381999999998, 2024-04-20T02:52:24.904 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.904 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.904 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.904 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.904 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 939 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406989+0000", 2024-04-20T02:52:24.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406993+0000", 2024-04-20T02:52:24.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9400000000000004e-06 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406999+0000", 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4389999999999999e-06 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407008+0000", 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3190000000000001e-06 2024-04-20T02:52:24.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.454982+0000", 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047973729999999999 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455051+0000", 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9197999999999997e-05 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455157+0000", 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106302 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456120+0000", 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096236599999999998 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456272+0000", 2024-04-20T02:52:24.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000152573 2024-04-20T02:52:24.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461274+0000", 2024-04-20T02:52:24.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005002055 2024-04-20T02:52:24.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461318+0000", 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3282999999999999e-05 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461370+0000", 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2704999999999998e-05 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.911 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:940 4.3 4:ca036f62:::smithi10548823-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.455002754000006, 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054543401999999998, 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 940 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:52:24.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407014+0000", 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407017+0000", 2024-04-20T02:52:24.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9790000000000002e-06 2024-04-20T02:52:24.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407022+0000", 2024-04-20T02:52:24.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5539999999999997e-06 2024-04-20T02:52:24.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407027+0000", 2024-04-20T02:52:24.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6099999999999999e-06 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455643+0000", 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.048616331999999998 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455714+0000", 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0610000000000003e-05 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461403+0000", 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056888219999999996 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461480+0000", 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7342000000000006e-05 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461558+0000", 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7780000000000004e-05 2024-04-20T02:52:24.916 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.317357219000002, 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:24.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 93.929434757999999, 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.921 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:52:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:52:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:52:24.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:52:24.926 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 93.929174552000006, 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.927 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:52:24.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:52:24.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 93.929153084000006, 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.933 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:52:24.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.934 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:24.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:24.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:52:24.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:52:24.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:52:24.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:52:24.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 84.735481152000006, 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:52:24.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:52:24.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:52:24.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:52:24.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:52:24.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:52:24.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:52:24.945 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.945 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.945 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.945 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:133 3.1 3:8efdf53d:::0188443c5b8894a8247535d52a547bece3a4c5f490fb220852b80e77d71ddcda:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:24.945 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 84.734756630000007, 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060974049000000002, 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:52:24.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127260+0000", 2024-04-20T02:52:24.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127435+0000", 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017459800000000001 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127443+0000", 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3790000000000006e-06 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127450+0000", 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5089999999999999e-06 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.129973+0000", 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025229639999999999 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130090+0000", 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117521 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130323+0000", 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023271199999999999 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.147581+0000", 2024-04-20T02:52:24.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017258440999999999 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155339+0000", 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077577189999999997 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188139+0000", 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032800425000000001 2024-04-20T02:52:24.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188195+0000", 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5773999999999998e-05 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188235+0000", 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9427e-05 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.952 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:136 3.1 3:a965252c:::a490375a0954219328244dfa32cf9967dfac8ed04667cb9b5ce15f4910db2ee6:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 84.734561412999994, 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060878320999999999, 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:52:24.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:52:24.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127455+0000", 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128197+0000", 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074176999999999999 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128208+0000", 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0498e-05 2024-04-20T02:52:24.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128216+0000", 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2630000000000003e-06 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135797+0000", 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0075808669999999998 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135930+0000", 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013299000000000001 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.136192+0000", 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026213899999999999 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148053+0000", 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011860723 2024-04-20T02:52:24.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155346+0000", 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0072930039999999996 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188270+0000", 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032924299999999997 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188298+0000", 2024-04-20T02:52:24.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8041000000000001e-05 2024-04-20T02:52:24.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188334+0000", 2024-04-20T02:52:24.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6247e-05 2024-04-20T02:52:24.959 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.959 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:139 3.1 3:90772484:::f9b8a83c02fab1c87f6aef9574ab68b1d6a75b657e100bac3693f85cac340cb2:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 84.727903487999995, 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054361675999999998, 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134113+0000", 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:24.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134645+0000", 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053281300000000004 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134662+0000", 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6503000000000001e-05 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134672+0000", 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8090000000000004e-06 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148073+0000", 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013401091 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148177+0000", 2024-04-20T02:52:24.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104344 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148417+0000", 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023968899999999999 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155385+0000", 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0069679260000000002 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155725+0000", 2024-04-20T02:52:24.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034046800000000002 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188383+0000", 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032658154000000002 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188433+0000", 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0059999999999998e-05 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188475+0000", 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2049999999999999e-05 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:140 3.1 3:b80bf598:::82930be3d6fd0d013eae1f4ab54a437986489fd06a4f4934c2432c14687252ab:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 84.727338232999998, 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053882434, 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134679+0000", 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134678+0000", 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134756+0000", 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7527999999999994e-05 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134763+0000", 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.092e-06 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134769+0000", 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0290000000000002e-06 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152310+0000", 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017540581999999999 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152431+0000", 2024-04-20T02:52:24.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012166 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.152621+0000", 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000189944 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155992+0000", 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033703470000000001 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.160642+0000", 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046505239999999996 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188508+0000", 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027866134000000001 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188534+0000", 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5945999999999999e-05 2024-04-20T02:52:24.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188561+0000", 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7300999999999998e-05 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1802 4.6 4:697da67d:::smithi10548823-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 70.782204440000001, 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053694407999999999, 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.972 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1802 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079813+0000", 2024-04-20T02:52:24.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079812+0000", 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079815+0000", 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1329999999999999e-06 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079823+0000", 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.401e-06 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.079828+0000", 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4149999999999998e-06 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124387+0000", 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044558788000000002 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.124468+0000", 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1447999999999997e-05 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133186+0000", 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0087175610000000004 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133344+0000", 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158393 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:14.133507+0000", 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000162829 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2292 4.3 4:c07c6daa:::smithi10548823-7:head [write 544593~799729] snapc 1e=[1e,1d,1b,18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.702437959999997, 2024-04-20T02:52:24.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061161747000000002, 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2292 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159577+0000", 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160019+0000", 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000442674 2024-04-20T02:52:24.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160045+0000", 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5446e-05 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160058+0000", 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3444e-05 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160487+0000", 2024-04-20T02:52:24.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000428899 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160616+0000", 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012861099999999999 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160838+0000", 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000222415 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207115+0000", 2024-04-20T02:52:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046276543000000003 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207164+0000", 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9497999999999999e-05 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220657+0000", 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013492386 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220698+0000", 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1451e-05 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220741+0000", 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3078000000000003e-05 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.983 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2699 4.0 e95/60 4:007b2a87:::smithi10548823-39:head v 95'100, mlcod=94'97)", 2024-04-20T02:52:24.984 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:24.984 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 32.512856671999998, 2024-04-20T02:52:24.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061625529999999998, 2024-04-20T02:52:24.984 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:24.984 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:24.984 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:24.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:24.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:24.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349160+0000", 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349351+0000", 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019061600000000001 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349364+0000", 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2835e-05 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349376+0000", 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349466+0000", 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9615000000000002e-05 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349492+0000", 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6494000000000001e-05 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.410234+0000", 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060741334000000001 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:24.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:24.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:24.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.410786+0000", 2024-04-20T02:52:24.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000552539 2024-04-20T02:52:24.988 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.988 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.988 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.988 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:24.988 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:24.988 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:24.990 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:24.991 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:24.990+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:25.091 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:25.092+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:25.170 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:25.170 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:25.171 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:25.171 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:25.180 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:25.192 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:25.193+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:25.271 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:52:25.279 INFO:tasks.rados.rados.0.smithi105.stdout:1514: done (6 left) 2024-04-20T02:52:25.279 INFO:tasks.rados.rados.0.smithi105.stdout:1515: done (5 left) 2024-04-20T02:52:25.279 INFO:tasks.rados.rados.0.smithi105.stdout:1517: done (4 left) 2024-04-20T02:52:25.279 INFO:tasks.rados.rados.0.smithi105.stdout:1518: done (3 left) 2024-04-20T02:52:25.280 INFO:tasks.rados.rados.0.smithi105.stdout:1520: done (2 left) 2024-04-20T02:52:25.280 INFO:tasks.rados.rados.0.smithi105.stdout:1521: done (1 left) 2024-04-20T02:52:25.280 INFO:tasks.rados.rados.0.smithi105.stdout:1522: done (0 left) 2024-04-20T02:52:25.280 INFO:tasks.rados.rados.0.smithi105.stdout:1523: set_chunk oid 4 target oid 2024-04-20T02:52:25.288 INFO:tasks.rados.rados.0.smithi105.stdout:1523: set_chunk oid 4 offset: 642048 length: 217088 target oid 7eca939cd9bc917dcc4f0994429ab295fbe8b23968d077ab5476eb21e090f438 offset: 0 snap -1 2024-04-20T02:52:25.288 INFO:tasks.rados.rados.0.smithi105.stdout:1524: read oid 22 snap -1 2024-04-20T02:52:25.288 INFO:tasks.rados.rados.0.smithi105.stdout:1524: expect (ObjNum 273 snap 26 seq_num 273) 2024-04-20T02:52:25.288 INFO:tasks.rados.rados.0.smithi105.stdout:1525: write (excl) oid 20 current snap is 30 2024-04-20T02:52:25.288 INFO:tasks.rados.rados.0.smithi105.stdout:1525: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-04-20T02:52:25.293 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:25.293+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:25.294 INFO:tasks.rados.rados.0.smithi105.stdout:1525: writing smithi10548823-20 from 457223 to 1025598 tid 1 2024-04-20T02:52:25.299 INFO:tasks.rados.rados.0.smithi105.stdout:1525: writing smithi10548823-20 from 1461649 to 1937407 tid 2 2024-04-20T02:52:25.306 INFO:tasks.rados.rados.0.smithi105.stdout:1525: writing smithi10548823-20 from 2420186 to 2857223 tid 3 2024-04-20T02:52:25.306 INFO:tasks.rados.rados.0.smithi105.stdout:1526: rollback oid 8 current snap is 30 2024-04-20T02:52:25.306 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 8 to 23 2024-04-20T02:52:25.306 INFO:tasks.rados.rados.0.smithi105.stdout:1523: finishing set_chunk to oid 4 2024-04-20T02:52:25.306 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 167 (ObjNum 191 snap 19 seq_num 191) dirty exists 2024-04-20T02:52:25.307 INFO:tasks.rados.rados.0.smithi105.stdout:1524: expect (ObjNum 273 snap 26 seq_num 273) 2024-04-20T02:52:25.325 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:25.325 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:25.325 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:25.326 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:25.326 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:25.340 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:25.424 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:25.424+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:25.486 INFO:tasks.rados.rados.0.smithi105.stdout:1526: finishing rollback tid 0 to smithi10548823-8 2024-04-20T02:52:25.486 INFO:tasks.rados.rados.0.smithi105.stdout:1526: finishing rollback tid 1 to smithi10548823-8 2024-04-20T02:52:25.486 INFO:tasks.rados.rados.0.smithi105.stdout:1523: done (3 left) 2024-04-20T02:52:25.486 INFO:tasks.rados.rados.0.smithi105.stdout:1524: done (2 left) 2024-04-20T02:52:25.486 INFO:tasks.rados.rados.0.smithi105.stdout:1527: snap_create 2024-04-20T02:52:25.487 INFO:tasks.rados.rados.0.smithi105.stdout:1526: finishing rollback tid 2 to smithi10548823-8 2024-04-20T02:52:25.487 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 117 (ObjNum 257 snap 22 seq_num 257) dirty exists 2024-04-20T02:52:25.487 INFO:tasks.rados.rados.0.smithi105.stdout:1525: finishing write tid 1 to smithi10548823-20 2024-04-20T02:52:25.487 INFO:tasks.rados.rados.0.smithi105.stdout:1525: finishing write tid 2 to smithi10548823-20 2024-04-20T02:52:25.487 INFO:tasks.rados.rados.0.smithi105.stdout:1525: finishing write tid 3 to smithi10548823-20 2024-04-20T02:52:25.487 INFO:tasks.rados.rados.0.smithi105.stdout:1525: finishing write tid 4 to smithi10548823-20 2024-04-20T02:52:25.488 INFO:tasks.rados.rados.0.smithi105.stdout:1525: finishing write tid 5 to smithi10548823-20 2024-04-20T02:52:25.488 INFO:tasks.rados.rados.0.smithi105.stdout:1525: oid 20 updating version 0 to 185 2024-04-20T02:52:25.488 INFO:tasks.rados.rados.0.smithi105.stdout:1525: oid 20 version 185 is already newer than 184 2024-04-20T02:52:25.488 INFO:tasks.rados.rados.0.smithi105.stdout:1525: oid 20 updating version 185 to 186 2024-04-20T02:52:25.488 INFO:tasks.rados.rados.0.smithi105.stdout:1525: oid 20 updating version 186 to 187 2024-04-20T02:52:25.488 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 187 (ObjNum 289 snap 30 seq_num 289) dirty exists 2024-04-20T02:52:25.488 INFO:tasks.rados.rados.0.smithi105.stdout:1525: left oid 20 (ObjNum 289 snap 30 seq_num 289) 2024-04-20T02:52:25.495 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:25.495+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:25.511 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:25.511 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.925744313999999, 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.512 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:52:25.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:52:25.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:52:25.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:52:25.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.909726035999995, 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.520 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.521 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:52:25.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.521 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.522 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.523 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:52:25.523 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:52:25.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.523 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.523 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:52:25.523 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:52:25.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.523 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:52:25.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:52:25.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:52:25.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:52:25.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:52:25.525 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:52:25.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:25.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:52:25.525 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:52:25.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:52:25.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:52:25.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:52:25.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:52:25.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:52:25.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:52:25.528 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.528 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.528 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.528 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:25.528 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:52:25.528 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.909572807000004, 2024-04-20T02:52:25.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:52:25.529 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.529 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.529 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.529 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.529 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.529 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:52:25.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.529 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.529 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:52:25.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:52:25.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:52:25.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:52:25.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:52:25.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:52:25.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:52:25.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:52:25.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:52:25.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:52:25.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.537 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:25.537 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:52:25.537 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.909462798000007, 2024-04-20T02:52:25.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:52:25.537 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.537 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:52:25.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:52:25.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:52:25.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:52:25.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:52:25.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:52:25.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:52:25.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:52:25.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.542 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:52:25.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:52:25.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:52:25.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:52:25.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:52:25.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:52:25.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:52:25.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:52:25.546 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.546 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.546 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.546 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:25.546 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:25.546 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.909432421999995, 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:25.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:25.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:52:25.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:52:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:52:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:52:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:52:25.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:52:25.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:52:25.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:52:25.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.905015003000003, 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:52:25.553 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:25.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:52:25.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:25.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:52:25.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:52:25.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:52:25.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:52:25.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:25.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:52:25.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:52:25.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:52:25.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:52:25.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:52:25.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:52:25.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:52:25.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:52:25.562 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.562 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.562 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.562 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.895763251999995, 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.563 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:52:25.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:25.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:52:25.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:52:25.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:52:25.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:52:25.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:52:25.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:52:25.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:52:25.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:52:25.570 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.570 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.570 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.571 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:25.571 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:52:25.572 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.893241109000002, 2024-04-20T02:52:25.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:52:25.572 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.572 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.573 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.573 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.573 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.574 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:52:25.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.574 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:52:25.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:52:25.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:52:25.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:25.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:52:25.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:52:25.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:52:25.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:52:25.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:52:25.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:52:25.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:52:25.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:52:25.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:52:25.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:52:25.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:25.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:52:25.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:52:25.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:52:25.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:52:25.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:52:25.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:52:25.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:52:25.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:52:25.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:52:25.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:52:25.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:52:25.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:52:25.586 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.586 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.586 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.587 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:25.587 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:25.587 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.893156242000003, 2024-04-20T02:52:25.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:52:25.588 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.588 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.588 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.588 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.589 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.589 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:52:25.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.589 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:25.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:25.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:25.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:25.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:52:25.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:52:25.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:52:25.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:52:25.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:52:25.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:52:25.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:25.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:52:25.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:52:25.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:52:25.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:52:25.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:52:25.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:52:25.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.893138632000003, 2024-04-20T02:52:25.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:52:25.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:52:25.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:52:25.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 94.543772024000006, 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058137664999999998, 2024-04-20T02:52:25.605 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967112+0000", 2024-04-20T02:52:25.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:25.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981895+0000", 2024-04-20T02:52:25.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014783522 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981906+0000", 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0560000000000001e-05 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981912+0000", 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981956+0000", 2024-04-20T02:52:25.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3455000000000003e-05 2024-04-20T02:52:25.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981979+0000", 2024-04-20T02:52:25.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3475000000000001e-05 2024-04-20T02:52:25.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025222+0000", 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043242622000000001 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025250+0000", 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8520000000000001e-05 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 91.098764036000006, 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:25.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:25.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:52:25.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:25.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:52:25.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:52:25.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:52:25.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:52:25.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:52:25.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:52:25.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:52:25.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:52:25.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:52:25.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:52:25.617 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.617 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.617 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.618 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:25.618 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:25.618 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 91.092759247999993, 2024-04-20T02:52:25.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:52:25.618 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.618 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.619 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:25.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:25.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:52:25.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:25.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:52:25.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:52:25.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:52:25.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:52:25.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:52:25.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:52:25.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:52:25.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:52:25.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:52:25.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:52:25.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:52:25.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:52:25.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:52:25.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:52:25.624 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.625 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.625 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.625 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:52:25.626 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:52:25.626 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.369543125, 2024-04-20T02:52:25.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056137746000000002, 2024-04-20T02:52:25.626 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.626 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.626 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:52:25.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141340+0000", 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142751+0000", 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014113540000000001 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142770+0000", 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8450000000000001e-05 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142780+0000", 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0284e-05 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142982+0000", 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000202244 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.143007+0000", 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4929000000000001e-05 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197446+0000", 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054438519999999997 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197479+0000", 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3597999999999999e-05 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2490 4.9 e90/81 4:9c87d36d:::smithi10548823-24:head v 90'246, mlcod=88'242)", 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.266444202000002, 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055338261999999999, 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.630 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:52:25.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244440+0000", 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245684+0000", 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001243755 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245696+0000", 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.171e-05 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245706+0000", 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0236e-05 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245756+0000", 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0571000000000001e-05 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245784+0000", 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7841999999999999e-05 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.299737+0000", 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053952420000000001 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.299779+0000", 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2014000000000001e-05 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2654 4.7 4:e9876a63:::smithi10548823-30:head [write 2075401~451067] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.040011030999999, 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063177293999999995, 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.637 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2654 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470871+0000", 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471109+0000", 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023773800000000001 2024-04-20T02:52:25.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471129+0000", 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0100999999999999e-05 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471139+0000", 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0427e-05 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.493743+0000", 2024-04-20T02:52:25.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02260334 2024-04-20T02:52:25.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.493897+0000", 2024-04-20T02:52:25.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015392800000000001 2024-04-20T02:52:25.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.494161+0000", 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026414899999999998 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510453+0000", 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016292054 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510509+0000", 2024-04-20T02:52:25.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6709000000000002e-05 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533920+0000", 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023410713999999999 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533989+0000", 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8421000000000001e-05 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534051+0000", 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2398999999999997e-05 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2655 4.7 4:e9876a63:::smithi10548823-30:head [write 3107008~648769] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.032108000000001, 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055363825999999998, 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.644 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2655 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478775+0000", 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478959+0000", 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018453600000000001 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478981+0000", 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1523e-05 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478991+0000", 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0288e-05 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510544+0000", 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031552795000000002 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510667+0000", 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012335699999999999 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510928+0000", 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000260487 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533864+0000", 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022936926 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533908+0000", 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3498000000000003e-05 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534060+0000", 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015223999999999999 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534113+0000", 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2612999999999999e-05 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534141+0000", 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7696e-05 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2656 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (57) in=7b,truncate 3798382] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.031877923000003, 2024-04-20T02:52:25.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055660077000000002, 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2656 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:52:25.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:52:25.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479006+0000", 2024-04-20T02:52:25.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479011+0000", 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.578e-06 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479015+0000", 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3869999999999997e-06 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479019+0000", 2024-04-20T02:52:25.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.703e-06 2024-04-20T02:52:25.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533236+0000", 2024-04-20T02:52:25.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054217551000000003 2024-04-20T02:52:25.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533355+0000", 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011906899999999999 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533595+0000", 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023979599999999999 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534575+0000", 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00097944500000000001 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534599+0000", 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4371000000000001e-05 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534605+0000", 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534631+0000", 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6469999999999999e-05 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534667+0000", 2024-04-20T02:52:25.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5822999999999999e-05 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2657 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.031862623999999, 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055802286, 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.658 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2657 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:25.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479024+0000", 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.779e-06 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479027+0000", 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4529999999999998e-06 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479030+0000", 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.993e-06 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534184+0000", 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055154305000000001 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534295+0000", 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110665 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534694+0000", 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039911500000000002 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534753+0000", 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8649e-05 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534824+0000", 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1589999999999997e-05 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2658 4.7 4:e2c0d65b:::smithi10548823-46:head [read 561152~397312] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 34.031852422999997, 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055503328999999997, 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2658 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479034+0000", 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5600000000000001e-06 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479036+0000", 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6280000000000001e-06 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479039+0000", 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5969999999999999e-06 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534331+0000", 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055292556 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534409+0000", 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7411999999999997e-05 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534536+0000", 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012682500000000001 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.667 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.668 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.668 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:25.668 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:25.668 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:25.668 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:25.670 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:25.670 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:25.670+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:25.770 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:25.771+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:25.844 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:25.844 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:25.845 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:25.845 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:25.846 INFO:teuthology.orchestra.run.smithi105.stdout: marking collection for removal 2024-04-20T02:52:25.846 INFO:teuthology.orchestra.run.smithi105.stdout:setting '_remove' omap key 2024-04-20T02:52:25.846 INFO:teuthology.orchestra.run.smithi105.stdout:finish_remove_pgs 4.16_head removing 4.16 2024-04-20T02:52:25.855 INFO:teuthology.orchestra.run.smithi105.stdout:Remove successful 2024-04-20T02:52:25.856 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:25.861 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:25.871 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:25.871+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:25.882 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:25.902 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:25.922 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:25.942 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:25.962 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:25.972 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:25.972+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:25.982 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:25.982 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:25.982 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:25.982 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T02:52:25.982 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:25.982 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:25.982 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:25.982 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:25.982 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:25.985 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:25.988 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:25.988 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:25.999 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:26.073 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:26.074+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:26.132 INFO:tasks.rados.rados.0.smithi105.stdout:1525: done (2 left) 2024-04-20T02:52:26.132 INFO:tasks.rados.rados.0.smithi105.stdout:1526: done (1 left) 2024-04-20T02:52:26.132 INFO:tasks.rados.rados.0.smithi105.stdout:1527: done (0 left) 2024-04-20T02:52:26.132 INFO:tasks.rados.rados.0.smithi105.stdout:1528: write (excl) oid 8 current snap is 31 2024-04-20T02:52:26.132 INFO:tasks.rados.rados.0.smithi105.stdout:1528: seq_num 290 ranges {474030=643624,1520131=735769} 2024-04-20T02:52:26.138 INFO:tasks.rados.rados.0.smithi105.stdout:1528: writing smithi10548823-8 from 474030 to 1117654 tid 1 2024-04-20T02:52:26.145 INFO:tasks.rados.rados.0.smithi105.stdout:1528: writing smithi10548823-8 from 1520131 to 2255900 tid 2 2024-04-20T02:52:26.145 INFO:tasks.rados.rados.0.smithi105.stdout:1529: read oid 9 snap -1 2024-04-20T02:52:26.145 INFO:tasks.rados.rados.0.smithi105.stdout:1529: expect deleted 2024-04-20T02:52:26.145 INFO:tasks.rados.rados.0.smithi105.stdout:1530: tier_promote oid 6 2024-04-20T02:52:26.145 INFO:tasks.rados.rados.0.smithi105.stdout:1531: snap_create 2024-04-20T02:52:26.149 INFO:tasks.rados.rados.0.smithi105.stdout:1530: got (0) Success 2024-04-20T02:52:26.149 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 178 (ObjNum 287 snap 29 seq_num 287) dirty exists 2024-04-20T02:52:26.174 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:26.175+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:26.174 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:26.174 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:26.185 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:26.197 INFO:tasks.rados.rados.0.smithi105.stdout:1528: finishing write tid 1 to smithi10548823-8 2024-04-20T02:52:26.200 INFO:tasks.rados.rados.0.smithi105.stdout:1528: finishing write tid 2 to smithi10548823-8 2024-04-20T02:52:26.200 INFO:tasks.rados.rados.0.smithi105.stdout:1528: finishing write tid 3 to smithi10548823-8 2024-04-20T02:52:26.200 INFO:tasks.rados.rados.0.smithi105.stdout:1528: finishing write tid 4 to smithi10548823-8 2024-04-20T02:52:26.200 INFO:tasks.rados.rados.0.smithi105.stdout:1528: oid 8 updating version 0 to 121 2024-04-20T02:52:26.200 INFO:tasks.rados.rados.0.smithi105.stdout:1528: oid 8 version 121 is already newer than 119 2024-04-20T02:52:26.200 INFO:tasks.rados.rados.0.smithi105.stdout:1528: oid 8 version 121 is already newer than 120 2024-04-20T02:52:26.200 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 121 (ObjNum 290 snap 31 seq_num 290) dirty exists 2024-04-20T02:52:26.201 INFO:tasks.rados.rados.0.smithi105.stdout:1528: left oid 8 (ObjNum 290 snap 31 seq_num 290) 2024-04-20T02:52:26.275 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:26.275+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:26.322 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:26.322 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:26.332 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:26.369 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.16.2 2024-04-20T02:52:26.416 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:26.417+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:26.476 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:26.477+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:26.502 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:26.502 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:26.512 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:26.596 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:26.597+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:26.680 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:26.680 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:26.680 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:26.681+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:26.690 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:26.781 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:26.782+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:26.861 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:26.861 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:26.870 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:26.912 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:26.911+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:26.983 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:26.983+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:27.038 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:27.039 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:27.049 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:27.093 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:27.093+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:27.135 INFO:tasks.rados.rados.0.smithi105.stdout:1528: done (3 left) 2024-04-20T02:52:27.135 INFO:tasks.rados.rados.0.smithi105.stdout:1529: done (2 left) 2024-04-20T02:52:27.135 INFO:tasks.rados.rados.0.smithi105.stdout:1530: done (1 left) 2024-04-20T02:52:27.135 INFO:tasks.rados.rados.0.smithi105.stdout:1531: done (0 left) 2024-04-20T02:52:27.135 INFO:tasks.rados.rados.0.smithi105.stdout:1532: write (excl) oid 6 current snap is 32 2024-04-20T02:52:27.135 INFO:tasks.rados.rados.0.smithi105.stdout:1532: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-04-20T02:52:27.141 INFO:tasks.rados.rados.0.smithi105.stdout:1532: writing smithi10548823-6 from 490837 to 1209710 tid 1 2024-04-20T02:52:27.147 INFO:tasks.rados.rados.0.smithi105.stdout:1532: writing smithi10548823-6 from 1862260 to 2341687 tid 2 2024-04-20T02:52:27.150 INFO:tasks.rados.rados.0.smithi105.stdout:1532: writing smithi10548823-6 from 2875032 to 2890837 tid 3 2024-04-20T02:52:27.150 INFO:tasks.rados.rados.0.smithi105.stdout:1533: snap_remove snap 23 2024-04-20T02:52:27.185 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:27.184+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:27.190 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:27.190 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:27.201 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:27.285 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:52:27.286 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:27.286+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:27.369 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:27.369 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:27.380 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:27.423 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:27.423+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:27.488 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:27.487+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:27.509 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:27.509 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:27.509 INFO:teuthology.orchestra.run.smithi105.stdout:get_pg_num_history pg_num_history pg_num_history(e121 pg_nums {1={14=1},2={18=8},3={21=4},4={22=16,60=26,81=25,85=24,89=23,96=26,110=25,114=24,118=23}} deleted_pools ) 2024-04-20T02:52:27.509 INFO:teuthology.orchestra.run.smithi105.stdout:Importing pgid 4.16 2024-04-20T02:52:27.513 INFO:teuthology.orchestra.run.smithi105.stdout:Write #4:68acf93d:::smithi10548823-22:32# 2024-04-20T02:52:27.519 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:27.543 INFO:teuthology.orchestra.run.smithi105.stdout:Write #4:68acf93d:::smithi10548823-22:head# 2024-04-20T02:52:27.543 INFO:teuthology.orchestra.run.smithi105.stdout:snapset 32=[]:{32=[2f,2e]} 2024-04-20T02:52:27.577 INFO:teuthology.orchestra.run.smithi105.stdout:Write #4:697da67d:::smithi10548823-48:head# 2024-04-20T02:52:27.578 INFO:teuthology.orchestra.run.smithi105.stdout:snapset 1e=[]:{} 2024-04-20T02:52:27.602 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:27.602+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:27.628 INFO:teuthology.orchestra.run.smithi105.stdout:write_pg epoch 121 info 4.16( v 117'91 (101'85,117'91] local-lis/les=120/121 n=3 ec=60/22 lis/c=120/120 les/c/f=121/121/0 sis=120) 2024-04-20T02:52:27.628 INFO:teuthology.orchestra.run.smithi105.stdout:Import successful 2024-04-20T02:52:27.688 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:27.688 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:27.689 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:27.689+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:27.699 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:27.789 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:27.789+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:27.869 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:27.869 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:27.879 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:27.890 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:27.891+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:27.986 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:28.031 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:28.031+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:28.047 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:28.048 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:28.058 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:28.101 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:28.101+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:28.130 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:28.137 INFO:tasks.rados.rados.0.smithi105.stdout:1533: done (1 left) 2024-04-20T02:52:28.137 INFO:tasks.rados.rados.0.smithi105.stdout:1534: write oid 22 current snap is 32 2024-04-20T02:52:28.137 INFO:tasks.rados.rados.0.smithi105.stdout:1534: seq_num 292 ranges {507644=794122,1920742=623085} 2024-04-20T02:52:28.145 INFO:tasks.rados.rados.0.smithi105.stdout:1534: writing smithi10548823-22 from 507644 to 1301766 tid 1 2024-04-20T02:52:28.153 INFO:tasks.rados.rados.0.smithi105.stdout:1534: writing smithi10548823-22 from 1920742 to 2543827 tid 2 2024-04-20T02:52:28.154 INFO:tasks.rados.rados.0.smithi105.stdout:1535: read oid 11 snap -1 2024-04-20T02:52:28.154 INFO:tasks.rados.rados.0.smithi105.stdout:1535: expect (ObjNum 282 snap 27 seq_num 282) 2024-04-20T02:52:28.154 INFO:tasks.rados.rados.0.smithi105.stdout:1536: set_chunk oid 13 target oid 2024-04-20T02:52:28.155 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /var/log/ceph/exp.4.16.2 2024-04-20T02:52:28.161 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:52:28.163 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:28.175 INFO:tasks.rados.rados.0.smithi105.stdout:1536: set_chunk oid 13 offset: 2119680 length: 246784 target oid 1a7107e5d420af43e3209de73db42f6c8f240343b170bdec87003e194fd5bcaa offset: 0 snap -1 2024-04-20T02:52:28.175 INFO:tasks.rados.rados.0.smithi105.stdout:1537: read oid 5 snap -1 2024-04-20T02:52:28.175 INFO:tasks.rados.rados.0.smithi105.stdout:1537: expect (ObjNum 262 snap 23 seq_num 262) 2024-04-20T02:52:28.175 INFO:tasks.rados.rados.0.smithi105.stdout:1538: delete oid 15 current snap is 32 2024-04-20T02:52:28.175 INFO:tasks.rados.rados.0.smithi105.stdout:1532: finishing write tid 1 to smithi10548823-6 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:1532: finishing write tid 2 to smithi10548823-6 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:1532: finishing write tid 3 to smithi10548823-6 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:1532: finishing write tid 4 to smithi10548823-6 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:1532: finishing write tid 5 to smithi10548823-6 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:1532: oid 6 updating version 0 to 193 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:1532: oid 6 updating version 193 to 194 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:1532: oid 6 version 194 is already newer than 192 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:1532: oid 6 version 194 is already newer than 191 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 194 (ObjNum 291 snap 32 seq_num 291) dirty exists 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:1532: left oid 6 (ObjNum 291 snap 32 seq_num 291) 2024-04-20T02:52:28.176 INFO:tasks.rados.rados.0.smithi105.stdout:1535: expect (ObjNum 282 snap 27 seq_num 282) 2024-04-20T02:52:28.195 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:28.194+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:28.197 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.2 2024-04-20T02:52:28.197 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-2 show-label 2024-04-20T02:52:28.200 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:28.201 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:28.212 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:28.284 INFO:teuthology.orchestra.run.smithi105.stdout:inferring bluefs devices from bluestore path 2024-04-20T02:52:28.284 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "/var/lib/ceph/osd/ceph-2/block": { 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_uuid": "336e852e-5fa6-4ac0-b053-d31ad40d7c4e", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 96636764160, 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "btime": "2024-04-20T02:49:38.198647+0000", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "main", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bfm_blocks": "23592960", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bfm_blocks_per_key": "128", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bfm_bytes_per_block": "4096", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bfm_size": "96636764160", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs": "1", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_fsid": "a524a365-eaec-4ebd-aa4d-0d76d55a252f", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version_when_created": "ceph version 18.2.2-1252-g15eecde2 (15eecde2bab9803889d929c7e423d07970d8cbed) reef (stable)", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "created_at": "2024-04-20T02:49:40.537169Z", 2024-04-20T02:52:28.285 INFO:teuthology.orchestra.run.smithi105.stdout: "kv_backend": "rocksdb", 2024-04-20T02:52:28.286 INFO:teuthology.orchestra.run.smithi105.stdout: "magic": "ceph osd volume v026", 2024-04-20T02:52:28.286 INFO:teuthology.orchestra.run.smithi105.stdout: "mkfs_done": "yes", 2024-04-20T02:52:28.286 INFO:teuthology.orchestra.run.smithi105.stdout: "ready": "ready", 2024-04-20T02:52:28.286 INFO:teuthology.orchestra.run.smithi105.stdout: "require_osd_release": "18", 2024-04-20T02:52:28.286 INFO:teuthology.orchestra.run.smithi105.stdout: "whoami": "2" 2024-04-20T02:52:28.286 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.286 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:28.287 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.2 2024-04-20T02:52:28.287 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-2 show-sharding 2024-04-20T02:52:28.328 INFO:tasks.rados.rados.0.smithi105.stdout:1536: finishing set_chunk to oid 13 2024-04-20T02:52:28.328 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 13 set_chunk returned error code -95 offset: 2119680 length: 246784 2024-04-20T02:52:28.328 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 0 (ObjNum 151 snap 10 seq_num 151) dirty exists 2024-04-20T02:52:28.328 INFO:tasks.rados.rados.0.smithi105.stdout:1532: done (5 left) 2024-04-20T02:52:28.328 INFO:tasks.rados.rados.0.smithi105.stdout:1535: done (4 left) 2024-04-20T02:52:28.328 INFO:tasks.rados.rados.0.smithi105.stdout:1536: done (3 left) 2024-04-20T02:52:28.328 INFO:tasks.rados.rados.0.smithi105.stdout:1538: done (2 left) 2024-04-20T02:52:28.328 INFO:tasks.rados.rados.0.smithi105.stdout:1539: set_chunk oid 6 target oid 2024-04-20T02:52:28.328 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:28.327+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:28.363 INFO:tasks.rados.rados.0.smithi105.stdout:1539: set_chunk oid 6 offset: 217088 length: 1091584 target oid 846cba5174352141e120943cc8afd1bf3ee66b71a544067a518dd28f4507a448 offset: 0 snap -1 2024-04-20T02:52:28.363 INFO:tasks.rados.rados.0.smithi105.stdout:1540: read oid 50 snap -1 2024-04-20T02:52:28.363 INFO:tasks.rados.rados.0.smithi105.stdout:1540: expect (ObjNum 285 snap 29 seq_num 285) 2024-04-20T02:52:28.363 INFO:tasks.rados.rados.0.smithi105.stdout:1541: set_chunk oid 16 target oid 2024-04-20T02:52:28.363 INFO:tasks.rados.rados.0.smithi105.stdout:1541: set_chunk oid 16 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:28.364 INFO:tasks.rados.rados.0.smithi105.stdout:1542: tier_evict oid 42 2024-04-20T02:52:28.365 INFO:tasks.rados.rados.0.smithi105.stdout:1542: tier_evict oid 42 snap -1 2024-04-20T02:52:28.365 INFO:tasks.rados.rados.0.smithi105.stdout:1543: write oid 25 current snap is 32 2024-04-20T02:52:28.365 INFO:tasks.rados.rados.0.smithi105.stdout:1543: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-04-20T02:52:28.368 INFO:tasks.rados.rados.0.smithi105.stdout:1543: writing smithi10548823-25 from 541258 to 1085878 tid 1 2024-04-20T02:52:28.376 INFO:tasks.rados.rados.0.smithi105.stdout:1543: writing smithi10548823-25 from 1637706 to 2264460 tid 2 2024-04-20T02:52:28.383 INFO:tasks.rados.rados.0.smithi105.stdout:1543: writing smithi10548823-25 from 2840948 to 2941258 tid 3 2024-04-20T02:52:28.383 INFO:tasks.rados.rados.0.smithi105.stdout:1544: write oid 7 current snap is 32 2024-04-20T02:52:28.383 INFO:tasks.rados.rados.0.smithi105.stdout:1544: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-04-20T02:52:28.385 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:28.385 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:28.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.385 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3418 4.b 4:da76a656:::smithi10548823-25:head [write 541258~544620 in=544620b] snapc 3a=[3a,39,38,36,35,33] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-04-20T02:52:28.385 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.370203+0000", 2024-04-20T02:52:28.385 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.015827482, 2024-04-20T02:52:28.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015854476999999999, 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "waiting for sub ops", 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3418 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.370203+0000", 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.370203+0000", 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.370200+0000", 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.370456+0000", 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025611399999999999 2024-04-20T02:52:28.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.370473+0000", 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6370000000000001e-05 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.370482+0000", 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6260000000000007e-06 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.370538+0000", 2024-04-20T02:52:28.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5373e-05 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.370650+0000", 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011270800000000001 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.370864+0000", 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021386800000000001 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3419 4.b 4.6a656e5b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.377867+0000", 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0081631299999999993, 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0082480580000000008, 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "queued for pg", 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3419 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.377867+0000", 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.377867+0000", 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.377864+0000", 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.378044+0000", 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000179815 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.378064+0000", 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0055999999999999e-05 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.378080+0000", 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.535e-05 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.392 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3420 4.b 4.6a656e5b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.381740+0000", 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0042898299999999997, 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0044186970000000001, 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "queued for pg", 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3420 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.393 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381740+0000", 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381740+0000", 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381739+0000", 2024-04-20T02:52:28.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:28.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381813+0000", 2024-04-20T02:52:28.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4297999999999999e-05 2024-04-20T02:52:28.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381827+0000", 2024-04-20T02:52:28.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3906e-05 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381837+0000", 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.9569999999999997e-06 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3421 4.b 4.6a656e5b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.381846+0000", 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0041838079999999998, 2024-04-20T02:52:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043433899999999999, 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "queued for pg", 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3421 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381846+0000", 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381846+0000", 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381846+0000", 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381850+0000", 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0749999999999999e-06 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381856+0000", 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.054e-06 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381862+0000", 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7049999999999996e-06 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3422 4.b 4.6a656e5b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-04-20T02:52:28.400 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.381866+0000", 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.0041636870000000001, 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043553180000000004, 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "queued for pg", 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3422 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381866+0000", 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381866+0000", 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381866+0000", 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381869+0000", 2024-04-20T02:52:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4439999999999998e-06 2024-04-20T02:52:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381874+0000", 2024-04-20T02:52:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6489999999999996e-06 2024-04-20T02:52:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.381879+0000", 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0930000000000002e-06 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop_reply(client.4431.0:3418 4.b e130/22)", 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.385033+0000", 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.00099733299999999999, 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012175249999999999, 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.404 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "queued for pg", 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.385033+0000", 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.385033+0000", 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.385032+0000", 2024-04-20T02:52:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.385044+0000", 2024-04-20T02:52:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2388000000000001e-05 2024-04-20T02:52:28.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.385056+0000", 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1778e-05 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.385064+0000", 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8310000000000001e-06 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.407 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:52:28.408 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 6 2024-04-20T02:52:28.408 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:28.408 INFO:tasks.rados.rados.0.smithi105.stdout:1544: writing smithi10548823-7 from 558065 to 1177934 tid 1 2024-04-20T02:52:28.408 INFO:tasks.rados.rados.0.smithi105.stdout:1544: writing smithi10548823-7 from 1579835 to 2066600 tid 2 2024-04-20T02:52:28.409 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:28.409 INFO:tasks.rados.rados.0.smithi105.stdout:1544: writing smithi10548823-7 from 2468371 to 2958065 tid 3 2024-04-20T02:52:28.409 INFO:tasks.rados.rados.0.smithi105.stdout:1545: read oid 49 snap -1 2024-04-20T02:52:28.409 INFO:tasks.rados.rados.0.smithi105.stdout:1545: expect (ObjNum 260 snap 22 seq_num 260) 2024-04-20T02:52:28.410 INFO:tasks.rados.rados.0.smithi105.stdout:1546: read oid 2 snap 29 2024-04-20T02:52:28.410 INFO:tasks.rados.rados.0.smithi105.stdout:1546: expect (ObjNum 272 snap 26 seq_num 272) 2024-04-20T02:52:28.410 INFO:tasks.rados.rados.0.smithi105.stdout:1547: snap_remove snap 27 2024-04-20T02:52:28.412 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:28.412 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:28.409+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:28.421 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:28.444 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:28.464 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:28.483 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:28.484 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:28.484 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:28.484 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T02:52:28.484 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:28.484 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:28.484 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:28.484 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:28.484 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:28.488 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:28.530 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:28.530+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:28.554 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:28.554 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:28.555 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:28.555 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:28.565 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:28.610 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:28.611+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:28.711 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:28.711+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:28.738 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:28.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:28.738 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:28.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.738 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:927 4.3 4:cf11ccc0:::smithi10548823-4:head [write 3121996~104915] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:28.738 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:52:28.738 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.35930942, 2024-04-20T02:52:28.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058986447999999997, 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 927 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377549+0000", 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377548+0000", 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377607+0000", 2024-04-20T02:52:28.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9413999999999997e-05 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377616+0000", 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4880000000000007e-06 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377621+0000", 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.682e-06 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407778+0000", 2024-04-20T02:52:28.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030156110999999999 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407866+0000", 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8645000000000003e-05 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407985+0000", 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000118874 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411741+0000", 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037563060000000001 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411854+0000", 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112898 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436438+0000", 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024583885 2024-04-20T02:52:28.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436483+0000", 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5334999999999999e-05 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436535+0000", 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1780000000000002e-05 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:928 4.3 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (55) in=7b,truncate 3226911] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.359230814, 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059064512999999999, 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 928 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377627+0000", 2024-04-20T02:52:28.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377630+0000", 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7259999999999998e-06 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377633+0000", 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1489999999999998e-06 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377636+0000", 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7760000000000002e-06 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410754+0000", 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033117944000000003 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.410850+0000", 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6522999999999997e-05 2024-04-20T02:52:28.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411008+0000", 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158157 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436151+0000", 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025142813 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436160+0000", 2024-04-20T02:52:28.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1460000000000002e-06 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436588+0000", 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00042776499999999998 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436631+0000", 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3084000000000003e-05 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436692+0000", 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0829000000000003e-05 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:929 4.3 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.359219673, 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.059214058, 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.751 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 929 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377639+0000", 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377638+0000", 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377640+0000", 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5430000000000001e-06 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377643+0000", 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0900000000000001e-06 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.377647+0000", 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2230000000000001e-06 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411271+0000", 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033623635999999998 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411329+0000", 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8043e-05 2024-04-20T02:52:28.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436728+0000", 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025399300999999999 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436793+0000", 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5041000000000003e-05 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436853+0000", 2024-04-20T02:52:28.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9446e-05 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:936 4.3 4:ca036f62:::smithi10548823-34:head [write 443718~615603] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.34218434, 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060880228000000002, 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.756 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 936 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394674+0000", 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394672+0000", 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394825+0000", 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015343299999999999 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394844+0000", 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8414999999999999e-05 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.394859+0000", 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4738e-05 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411879+0000", 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017020469999999999 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.411954+0000", 2024-04-20T02:52:28.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5239000000000002e-05 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.412104+0000", 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014944399999999999 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436402+0000", 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.024297828 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436409+0000", 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1779999999999997e-06 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455390+0000", 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018981221 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455508+0000", 2024-04-20T02:52:28.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117556 2024-04-20T02:52:28.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455554+0000", 2024-04-20T02:52:28.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6570000000000003e-05 2024-04-20T02:52:28.762 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.762 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:937 4.3 4:ca036f62:::smithi10548823-34:head [write 1564138~470341] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.33684601, 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060119655000000001, 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.763 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.764 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.764 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 937 2024-04-20T02:52:28.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.764 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:52:28.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400012+0000", 2024-04-20T02:52:28.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400010+0000", 2024-04-20T02:52:28.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:28.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400222+0000", 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000211208 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400238+0000", 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419e-05 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.400250+0000", 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2091e-05 2024-04-20T02:52:28.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425851+0000", 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025600795999999999 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.425965+0000", 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011411799999999999 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:28.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.426134+0000", 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168517 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436937+0000", 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01080328 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455351+0000", 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018414607 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460001+0000", 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046491809999999996 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460086+0000", 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5066000000000002e-05 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.460132+0000", 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6174999999999998e-05 2024-04-20T02:52:28.769 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:938 4.3 4:ca036f62:::smithi10548823-34:head [write 2633066~610652] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.33008055400001, 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054360654000000001, 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.770 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 938 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406778+0000", 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406775+0000", 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406947+0000", 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171619 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406962+0000", 2024-04-20T02:52:28.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5248999999999999e-05 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406979+0000", 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7334999999999999e-05 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.436954+0000", 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029974218 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437007+0000", 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3553e-05 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.437110+0000", 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455605+0000", 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.018494295000000001 2024-04-20T02:52:28.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455829+0000", 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022443100000000001 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461044+0000", 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052150499999999997 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461090+0000", 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6100999999999998e-05 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461138+0000", 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8003000000000001e-05 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:939 4.3 4:ca036f62:::smithi10548823-34:head [setxattr _header (56) in=7b,truncate 3243718] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:52:28.776 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.32986824300001, 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054380381999999998, 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 939 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:52:28.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406990+0000", 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406989+0000", 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406993+0000", 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9400000000000004e-06 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.406999+0000", 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4389999999999999e-06 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407008+0000", 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3190000000000001e-06 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.454982+0000", 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047973729999999999 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455051+0000", 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9197999999999997e-05 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455157+0000", 2024-04-20T02:52:28.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106302 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456120+0000", 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00096236599999999998 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.456272+0000", 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000152573 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461274+0000", 2024-04-20T02:52:28.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005002055 2024-04-20T02:52:28.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461318+0000", 2024-04-20T02:52:28.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3282999999999999e-05 2024-04-20T02:52:28.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461370+0000", 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2704999999999998e-05 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:940 4.3 4:ca036f62:::smithi10548823-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.329843694, 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054543401999999998, 2024-04-20T02:52:28.783 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 940 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407015+0000", 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407014+0000", 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407017+0000", 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9790000000000002e-06 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407022+0000", 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5539999999999997e-06 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.407027+0000", 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6099999999999999e-06 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455643+0000", 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.048616331999999998 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.455714+0000", 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0610000000000003e-05 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461403+0000", 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056888219999999996 2024-04-20T02:52:28.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461480+0000", 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7342000000000006e-05 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.461558+0000", 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7780000000000004e-05 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.192198159, 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:52:28.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:52:28.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:52:28.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 97.804275697999998, 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.793 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:52:28.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 97.804015492000005, 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.799 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:52:28.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:52:28.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:52:28.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.804 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 97.803994024000005, 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:52:28.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:52:28.809 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 88.610322092000004, 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:52:28.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:133 3.1 3:8efdf53d:::0188443c5b8894a8247535d52a547bece3a4c5f490fb220852b80e77d71ddcda:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 88.609597570000005, 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060974049000000002, 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127261+0000", 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127260+0000", 2024-04-20T02:52:28.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127435+0000", 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017459800000000001 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127443+0000", 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3790000000000006e-06 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127450+0000", 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5089999999999999e-06 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.129973+0000", 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0025229639999999999 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130090+0000", 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117521 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.130323+0000", 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023271199999999999 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.147581+0000", 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017258440999999999 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155339+0000", 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0077577189999999997 2024-04-20T02:52:28.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188139+0000", 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032800425000000001 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188195+0000", 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5773999999999998e-05 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188235+0000", 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9427e-05 2024-04-20T02:52:28.821 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:136 3.1 3:a965252c:::a490375a0954219328244dfa32cf9967dfac8ed04667cb9b5ce15f4910db2ee6:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 88.609402352999993, 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060878320999999999, 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127456+0000", 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127455+0000", 2024-04-20T02:52:28.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128197+0000", 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074176999999999999 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128208+0000", 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0498e-05 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128216+0000", 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2630000000000003e-06 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135797+0000", 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0075808669999999998 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.135930+0000", 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013299000000000001 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.136192+0000", 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026213899999999999 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148053+0000", 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011860723 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155346+0000", 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0072930039999999996 2024-04-20T02:52:28.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188270+0000", 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032924299999999997 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188298+0000", 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8041000000000001e-05 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188334+0000", 2024-04-20T02:52:28.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6247e-05 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:139 3.1 3:90772484:::f9b8a83c02fab1c87f6aef9574ab68b1d6a75b657e100bac3693f85cac340cb2:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 88.602744427999994, 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054361675999999998, 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134114+0000", 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134113+0000", 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134645+0000", 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053281300000000004 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134662+0000", 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6503000000000001e-05 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.134672+0000", 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8090000000000004e-06 2024-04-20T02:52:28.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148073+0000", 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013401091 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148177+0000", 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000104344 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.148417+0000", 2024-04-20T02:52:28.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023968899999999999 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155385+0000", 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0069679260000000002 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155725+0000", 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034046800000000002 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188383+0000", 2024-04-20T02:52:28.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032658154000000002 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188433+0000", 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0059999999999998e-05 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188475+0000", 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2049999999999999e-05 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2292 4.3 4:c07c6daa:::smithi10548823-7:head [write 544593~799729] snapc 1e=[1e,1d,1b,18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 54.577278900000003, 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061161747000000002, 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2292 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159577+0000", 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:28.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160019+0000", 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000442674 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160045+0000", 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5446e-05 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160058+0000", 2024-04-20T02:52:28.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3444e-05 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160487+0000", 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000428899 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160616+0000", 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012861099999999999 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:28.837 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160838+0000", 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000222415 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207115+0000", 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046276543000000003 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207164+0000", 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9497999999999999e-05 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.838 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220657+0000", 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013492386 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220698+0000", 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1451e-05 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220741+0000", 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3078000000000003e-05 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.839 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2699 4.0 e95/60 4:007b2a87:::smithi10548823-39:head v 95'100, mlcod=94'97)", 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 36.387697611999997, 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061625529999999998, 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:28.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349160+0000", 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349351+0000", 2024-04-20T02:52:28.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019061600000000001 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349364+0000", 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2835e-05 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349376+0000", 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349466+0000", 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9615000000000002e-05 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349492+0000", 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6494000000000001e-05 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.410234+0000", 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060741334000000001 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.410786+0000", 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000552539 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3423 4.3 4:c07c6daa:::smithi10548823-7:head [write 558065~619869] snapc 3a=[3a,39,38,36,35,33] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.34740987699999998, 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061409060000000001, 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.844 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3423 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:52:28.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389446+0000", 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389586+0000", 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014017899999999999 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389605+0000", 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389617+0000", 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1956999999999999e-05 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.412832+0000", 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023215612 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.412909+0000", 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6555000000000001e-05 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.413035+0000", 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000126599 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450700+0000", 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037664717 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450712+0000", 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1422e-05 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450741+0000", 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9791999999999999e-05 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450801+0000", 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9199e-05 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.849 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450857+0000", 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6882999999999997e-05 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3424 4.3 4:c07c6daa:::smithi10548823-7:head [write 1579835~486765] snapc 3a=[3a,39,38,36,35,33] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.395015+0000", 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 0.34184311499999998, 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056413509000000001, 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:28.850 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3424 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.395015+0000", 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.395015+0000", 2024-04-20T02:52:28.851 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.395014+0000", 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.395234+0000", 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022036399999999999 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.395251+0000", 2024-04-20T02:52:28.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6886999999999999e-05 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.395265+0000", 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4439e-05 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.427754+0000", 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032489166 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:28.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.427853+0000", 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8271999999999997e-05 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.428012+0000", 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015902900000000001 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450906+0000", 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022894181 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450911+0000", 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1329999999999998e-06 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.451297+0000", 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038635299999999998 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.451386+0000", 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8457999999999999e-05 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:28.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.451429+0000", 2024-04-20T02:52:28.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2796000000000002e-05 2024-04-20T02:52:28.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.856 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:28.856 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:28.856 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:28.858 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:28.858 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:28.857+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:28.958 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:28.958+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:29.027 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:29.027 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:29.027 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:29.027 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:29.037 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:29.078 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:29.078+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:29.144 INFO:teuthology.orchestra.run.smithi105.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-04-20T02:52:29.148 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.2 2024-04-20T02:52:29.148 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-04-20T02:52:29.149 INFO:tasks.thrashosds.thrasher:applying new sharding: O(5,5-) P(4,5-35) 2024-04-20T02:52:29.149 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-2 --sharding 'O(5,5-) P(4,5-35)' reshard 2024-04-20T02:52:29.160 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:29.159+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:29.210 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:29.210 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:29.211 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:29.221 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:29.261 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:29.260+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:29.286 INFO:tasks.rados.rados.0.smithi105.stdout:1547: done (10 left) 2024-04-20T02:52:29.286 INFO:tasks.rados.rados.0.smithi105.stdout:1548: tier_flush oid 29 2024-04-20T02:52:29.286 INFO:tasks.rados.rados.0.smithi105.stdout:1548: tier_flush oid 29 snap -1 2024-04-20T02:52:29.287 INFO:tasks.rados.rados.0.smithi105.stdout:1549: write oid 18 current snap is 32 2024-04-20T02:52:29.287 INFO:tasks.rados.rados.0.smithi105.stdout:1537: expect (ObjNum 262 snap 23 seq_num 262) 2024-04-20T02:52:29.362 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:29.362+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:29.370 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:29.370 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:29.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:29.370 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:29.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.370 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.370 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:29.370 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.78416471200001, 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:52:29.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:52:29.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.768146434, 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:52:29.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:52:29.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.382 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.767993205, 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.383 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.384 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:52:29.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.384 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:52:29.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.767883196, 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.390 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:52:29.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:52:29.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.76785282, 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:29.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.763435401, 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.401 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:29.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:52:29.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:52:29.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.75418365, 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:52:29.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:52:29.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.75166150699999, 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.414 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:52:29.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.75157664, 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.420 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:52:29.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:52:29.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.426 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.75155903, 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:29.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:52:29.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:52:29.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.431 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.402192421999999, 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058137664999999998, 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:52:29.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967112+0000", 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981895+0000", 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014783522 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981906+0000", 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0560000000000001e-05 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981912+0000", 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981956+0000", 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3455000000000003e-05 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981979+0000", 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3475000000000001e-05 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025222+0000", 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043242622000000001 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025250+0000", 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8520000000000001e-05 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 94.957184433999998, 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.436 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:52:29.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 94.951179646, 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.227963523, 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056137746000000002, 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.445 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141340+0000", 2024-04-20T02:52:29.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142751+0000", 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014113540000000001 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142770+0000", 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8450000000000001e-05 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142780+0000", 2024-04-20T02:52:29.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0284e-05 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142982+0000", 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000202244 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.143007+0000", 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4929000000000001e-05 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197446+0000", 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054438519999999997 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197479+0000", 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3597999999999999e-05 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2490 4.9 e90/81 4:9c87d36d:::smithi10548823-24:head v 90'246, mlcod=88'242)", 2024-04-20T02:52:29.449 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.124864600000002, 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055338261999999999, 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244440+0000", 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245684+0000", 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001243755 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245696+0000", 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.171e-05 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245706+0000", 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0236e-05 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245756+0000", 2024-04-20T02:52:29.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0571000000000001e-05 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245784+0000", 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7841999999999999e-05 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.299737+0000", 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053952420000000001 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.299779+0000", 2024-04-20T02:52:29.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2014000000000001e-05 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2654 4.7 4:e9876a63:::smithi10548823-30:head [write 2075401~451067] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.898431428999999, 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063177293999999995, 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.454 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2654 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470871+0000", 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471109+0000", 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023773800000000001 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471129+0000", 2024-04-20T02:52:29.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0100999999999999e-05 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471139+0000", 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0427e-05 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.493743+0000", 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02260334 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.493897+0000", 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015392800000000001 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.494161+0000", 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026414899999999998 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510453+0000", 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016292054 2024-04-20T02:52:29.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510509+0000", 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6709000000000002e-05 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533920+0000", 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023410713999999999 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533989+0000", 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8421000000000001e-05 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534051+0000", 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2398999999999997e-05 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2655 4.7 4:e9876a63:::smithi10548823-30:head [write 3107008~648769] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:29.460 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.890528398000001, 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055363825999999998, 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2655 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:52:29.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478775+0000", 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478959+0000", 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018453600000000001 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478981+0000", 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1523e-05 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478991+0000", 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0288e-05 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510544+0000", 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031552795000000002 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510667+0000", 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012335699999999999 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510928+0000", 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000260487 2024-04-20T02:52:29.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533864+0000", 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022936926 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533908+0000", 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3498000000000003e-05 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534060+0000", 2024-04-20T02:52:29.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015223999999999999 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534113+0000", 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2612999999999999e-05 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534141+0000", 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7696e-05 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.466 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2656 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (57) in=7b,truncate 3798382] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.890298321000003, 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055660077000000002, 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2656 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.467 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479006+0000", 2024-04-20T02:52:29.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479011+0000", 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.578e-06 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479015+0000", 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3869999999999997e-06 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479019+0000", 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.703e-06 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533236+0000", 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054217551000000003 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533355+0000", 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011906899999999999 2024-04-20T02:52:29.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533595+0000", 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023979599999999999 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534575+0000", 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00097944500000000001 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534599+0000", 2024-04-20T02:52:29.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4371000000000001e-05 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534605+0000", 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534631+0000", 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6469999999999999e-05 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534667+0000", 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5822999999999999e-05 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2657 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.890283021999998, 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055802286, 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:29.473 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2657 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479024+0000", 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.779e-06 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479027+0000", 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4529999999999998e-06 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479030+0000", 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.993e-06 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534184+0000", 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055154305000000001 2024-04-20T02:52:29.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534295+0000", 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110665 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534694+0000", 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039911500000000002 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534753+0000", 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8649e-05 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534824+0000", 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1589999999999997e-05 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2658 4.7 4:e2c0d65b:::smithi10548823-46:head [read 561152~397312] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:29.478 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.890272821000003, 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055503328999999997, 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2658 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:29.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479034+0000", 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5600000000000001e-06 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479036+0000", 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6280000000000001e-06 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479039+0000", 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5969999999999999e-06 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534331+0000", 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055292556 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534409+0000", 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7411999999999997e-05 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534536+0000", 2024-04-20T02:52:29.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012682500000000001 2024-04-20T02:52:29.483 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.483 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.483 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.483 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.483 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.483 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:29.484 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:29.484+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:29.485 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:29.486 INFO:tasks.rados.rados.0.smithi105.stdout:1534: finishing write tid 1 to smithi10548823-22 2024-04-20T02:52:29.486 INFO:tasks.rados.rados.0.smithi105.stdout:1534: finishing write tid 2 to smithi10548823-22 2024-04-20T02:52:29.486 INFO:tasks.rados.rados.0.smithi105.stdout:1534: finishing write tid 3 to smithi10548823-22 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:1534: finishing write tid 4 to smithi10548823-22 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:1534: oid 22 updating version 0 to 98 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:1534: oid 22 updating version 98 to 100 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:1534: oid 22 version 100 is already newer than 99 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 100 (ObjNum 292 snap 32 seq_num 292) dirty exists 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:1534: left oid 22 (ObjNum 292 snap 32 seq_num 292) 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:1541: finishing set_chunk to oid 16 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:1541: got expected ENOENT (src dne) 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:1542: got (22) Invalid argument 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:1539: finishing set_chunk to oid 6 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 194 (ObjNum 291 snap 32 seq_num 291) dirty exists 2024-04-20T02:52:29.487 INFO:tasks.rados.rados.0.smithi105.stdout:1540: expect (ObjNum 285 snap 29 seq_num 285) 2024-04-20T02:52:29.547 INFO:teuthology.orchestra.run.smithi105.stdout:reshard success 2024-04-20T02:52:29.585 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:29.585+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:29.616 INFO:tasks.rados.rados.0.smithi105.stdout:1545: expect (ObjNum 260 snap 22 seq_num 260) 2024-04-20T02:52:29.616 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:29.616 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:29.616 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:29.616 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:29.627 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:29.686 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:29.687+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:29.757 INFO:tasks.rados.rados.0.smithi105.stdout:1543: finishing write tid 1 to smithi10548823-25 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:1543: finishing write tid 2 to smithi10548823-25 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:1543: finishing write tid 3 to smithi10548823-25 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:1543: finishing write tid 4 to smithi10548823-25 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:1543: finishing write tid 5 to smithi10548823-25 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:1543: oid 25 updating version 0 to 204 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:1543: oid 25 updating version 204 to 206 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:1543: oid 25 version 206 is already newer than 205 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:1543: oid 25 version 206 is already newer than 203 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 206 (ObjNum 294 snap 32 seq_num 294) dirty exists 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:1543: left oid 25 (ObjNum 294 snap 32 seq_num 294) 2024-04-20T02:52:29.758 INFO:tasks.rados.rados.0.smithi105.stdout:1546: expect (ObjNum 272 snap 26 seq_num 272) 2024-04-20T02:52:29.759 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:29.759 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:29.759 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:29.759 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:29.759 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:29.768 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:29.787 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:29.788+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:29.888 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:29.888+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:29.927 INFO:tasks.rados.rados.0.smithi105.stdout:1544: finishing write tid 1 to smithi10548823-7 2024-04-20T02:52:29.927 INFO:tasks.rados.rados.0.smithi105.stdout:1544: finishing write tid 2 to smithi10548823-7 2024-04-20T02:52:29.927 INFO:tasks.rados.rados.0.smithi105.stdout:1549: seq_num 296 ranges {574872=695118,2038317=630423} 2024-04-20T02:52:29.933 INFO:tasks.rados.rados.0.smithi105.stdout:1549: writing smithi10548823-18 from 574872 to 1269990 tid 1 2024-04-20T02:52:29.939 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:29.939 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:29.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:29.939 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 103.394338976, 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:29.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:52:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.945 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 103.388117902, 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:52:29.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:52:29.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:52:29.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.951 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 103.37971769799999, 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.952 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:52:29.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:52:29.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:52:29.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:52:29.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:52:29.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:52:29.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:52:29.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:52:29.959 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.959 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.959 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.960 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:29.960 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:29.960 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 103.379375053, 2024-04-20T02:52:29.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:52:29.960 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.960 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:29.960 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.960 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.960 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.960 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:52:29.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 95.573053049999999, 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.965 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:52:29.970 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 95.565790088, 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.976 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 95.559847559000005, 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.977 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:52:29.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:52:29.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:52:29.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:52:29.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:52:29.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:52:29.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:52:29.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:52:29.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 95.559644567000007, 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.983 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:52:29.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 95.559621640000003, 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.989 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:52:29.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 95.549537905999998, 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.994 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:52:29.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:29.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:29.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 95.532652498000004, 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:29.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:52:30.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 95.516376057000002, 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:52:30.003 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:52:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 80.491820584999999, 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:52:30.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:52:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:52:30.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.014 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:52:30.014 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:30.014 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.177504808999998, 2024-04-20T02:52:30.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:52:30.014 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:30.014 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:30.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:30.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.015 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:52:30.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.015 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:30.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:30.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:30.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:30.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:30.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:30.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:52:30.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:30.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:30.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:52:30.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:52:30.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:30.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:52:30.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:52:30.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:52:30.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:52:30.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:52:30.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:52:30.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:30.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:52:30.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:52:30.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:30.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:52:30.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:52:30.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:52:30.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:52:30.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:52:30.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:52:30.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:30.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:52:30.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:52:30.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:30.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:52:30.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:52:30.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:30.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:52:30.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:52:30.023 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.023 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.024 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.024 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2545 4.9 e91/81 4:95e19a88:::smithi10548823-40:head v 91'253, mlcod=91'251)", 2024-04-20T02:52:30.024 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:30.024 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.522867693999999, 2024-04-20T02:52:30.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054757404000000003, 2024-04-20T02:52:30.024 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:30.024 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:30.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415210+0000", 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416004+0000", 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079381199999999997 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416022+0000", 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7969e-05 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416031+0000", 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9279999999999999e-06 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416100+0000", 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8891000000000002e-05 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416128+0000", 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8203000000000001e-05 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469912+0000", 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053783794000000003 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469968+0000", 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6474000000000002e-05 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:30.028 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:30.029 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.589838886999999, 2024-04-20T02:52:30.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:52:30.029 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:30.029 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.029 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:30.029 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:30.029 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.029 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:52:30.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:52:30.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:52:30.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2700 4.0 4:007b2a87:::smithi10548823-39:head [write 1967200~714747] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.581348693000002, 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061846287, 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.035 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2700 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356728+0000", 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356879+0000", 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000150958 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356894+0000", 2024-04-20T02:52:30.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5132e-05 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356906+0000", 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2071e-05 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366666+0000", 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0097595289999999994 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366774+0000", 2024-04-20T02:52:30.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108527 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366956+0000", 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181359 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390515+0000", 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023559902000000001 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406790+0000", 2024-04-20T02:52:30.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016274555999999999 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418488+0000", 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011697591 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418529+0000", 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1545000000000003e-05 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418576+0000", 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6903000000000001e-05 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2701 4.0 4:007b2a87:::smithi10548823-39:head [write 3421983~426820] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.574172701000002, 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054799176999999998, 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:30.041 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2701 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363904+0000", 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364154+0000", 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000250179 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364171+0000", 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6748e-05 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364182+0000", 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1371e-05 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390552+0000", 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026369607 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390652+0000", 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010059499999999999 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390857+0000", 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020445199999999999 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418224+0000", 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027367316999999999 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418282+0000", 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8279000000000002e-05 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418618+0000", 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000335967 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418661+0000", 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2614999999999998e-05 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418705+0000", 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3993999999999999e-05 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2702 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (57) in=7b,truncate 3848803] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.573886647000002, 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058669165000000002, 2024-04-20T02:52:30.047 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2702 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:30.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364191+0000", 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364196+0000", 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4490000000000001e-06 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:30.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364201+0000", 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5040000000000002e-06 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364206+0000", 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.405895+0000", 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041689047 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406021+0000", 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000125561 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406541+0000", 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052000699999999996 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418451+0000", 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011910041999999999 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418747+0000", 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029622399999999999 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422704+0000", 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039571859999999997 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422781+0000", 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.661e-05 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422861+0000", 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9987999999999994e-05 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2703 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 37.573867595000003, 2024-04-20T02:52:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058940278999999998, 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2703 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364213+0000", 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4830000000000002e-06 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364220+0000", 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.007e-06 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364225+0000", 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0350000000000001e-06 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406820+0000", 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042595135999999999 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406923+0000", 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102846 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422897+0000", 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015974202999999999 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423015+0000", 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117581 2024-04-20T02:52:30.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:30.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:30.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:30.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423151+0000", 2024-04-20T02:52:30.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013635899999999999 2024-04-20T02:52:30.058 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.058 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.058 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.058 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:30.058 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:30.058 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1549: writing smithi10548823-18 from 2038317 to 2668740 tid 2 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1534: done (11 left) 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1537: done (10 left) 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1539: done (9 left) 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1540: done (8 left) 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1541: done (7 left) 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1542: done (6 left) 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1543: done (5 left) 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1545: done (4 left) 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1546: done (3 left) 2024-04-20T02:52:30.060 INFO:tasks.rados.rados.0.smithi105.stdout:1550: write oid 45 current snap is 32 2024-04-20T02:52:30.061 INFO:tasks.rados.rados.0.smithi105.stdout:1544: finishing write tid 3 to smithi10548823-7 2024-04-20T02:52:30.061 INFO:tasks.rados.rados.0.smithi105.stdout:1544: finishing write tid 4 to smithi10548823-7 2024-04-20T02:52:30.061 INFO:tasks.rados.rados.0.smithi105.stdout:1550: seq_num 297 ranges {591679=486720,1813152=490434} 2024-04-20T02:52:30.061 INFO:tasks.rados.rados.0.smithi105.stdout:1550: writing smithi10548823-45 from 591679 to 1078399 tid 1 2024-04-20T02:52:30.062 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:30.062 INFO:tasks.rados.rados.0.smithi105.stdout:1550: writing smithi10548823-45 from 1813152 to 2303586 tid 2 2024-04-20T02:52:30.062 INFO:tasks.rados.rados.0.smithi105.stdout:1551: set_chunk oid 4 target oid 2024-04-20T02:52:30.063 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.2 2024-04-20T02:52:30.063 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-2 fsck 2024-04-20T02:52:30.067 INFO:tasks.rados.rados.0.smithi105.stdout:1551: set_chunk oid 4 offset: 1086464 length: 1455104 target oid 04d1fbd02c6f0a4c746d84e333363890fc1f4a52bf24f25fbaf249b29676dae8 offset: 0 snap -1 2024-04-20T02:52:30.067 INFO:tasks.rados.rados.0.smithi105.stdout:1552: read oid 38 snap -1 2024-04-20T02:52:30.067 INFO:tasks.rados.rados.0.smithi105.stdout:1552: expect deleted 2024-04-20T02:52:30.067 INFO:tasks.rados.rados.0.smithi105.stdout:1553: rollback oid 10 current snap is 32 2024-04-20T02:52:30.067 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 10 to 30 2024-04-20T02:52:30.067 INFO:tasks.rados.rados.0.smithi105.stdout:1544: finishing write tid 5 to smithi10548823-7 2024-04-20T02:52:30.067 INFO:tasks.rados.rados.0.smithi105.stdout:1544: oid 7 updating version 0 to 209 2024-04-20T02:52:30.067 INFO:tasks.rados.rados.0.smithi105.stdout:1544: oid 7 updating version 209 to 211 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1544: oid 7 updating version 211 to 212 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1544: oid 7 version 212 is already newer than 210 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 212 (ObjNum 295 snap 32 seq_num 295) dirty exists 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1544: left oid 7 (ObjNum 295 snap 32 seq_num 295) 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1548: got (0) Success 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 269 snap 25 seq_num 269) dirty exists 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1550: finishing write tid 1 to smithi10548823-45 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1550: finishing write tid 2 to smithi10548823-45 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1550: finishing write tid 3 to smithi10548823-45 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1544: done (6 left) 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1548: done (5 left) 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1554: read oid 42 snap -1 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1554: expect deleted 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1555: write (excl) oid 13 current snap is 32 2024-04-20T02:52:30.068 INFO:tasks.rados.rados.0.smithi105.stdout:1550: finishing write tid 4 to smithi10548823-45 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1550: oid 45 updating version 0 to 196 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1550: oid 45 updating version 196 to 197 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1550: oid 45 updating version 197 to 198 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 198 (ObjNum 297 snap 32 seq_num 297) dirty exists 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1550: left oid 45 (ObjNum 297 snap 32 seq_num 297) 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1549: finishing write tid 1 to smithi10548823-18 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1555: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1555: writing smithi10548823-13 from 608486 to 1170455 tid 1 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1555: writing smithi10548823-13 from 1871634 to 2505726 tid 2 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1555: writing smithi10548823-13 from 2950640 to 3008486 tid 3 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1550: done (6 left) 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1556: rollback oid 35 current snap is 32 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 35 to 31 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1549: finishing write tid 2 to smithi10548823-18 2024-04-20T02:52:30.069 INFO:tasks.rados.rados.0.smithi105.stdout:1549: finishing write tid 3 to smithi10548823-18 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1549: finishing write tid 4 to smithi10548823-18 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1549: oid 18 updating version 0 to 100 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1549: oid 18 version 100 is already newer than 98 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1549: oid 18 version 100 is already newer than 99 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 100 (ObjNum 296 snap 32 seq_num 296) dirty exists 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1549: left oid 18 (ObjNum 296 snap 32 seq_num 296) 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1549: done (6 left) 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1557: set_chunk oid 17 target oid 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1557: set_chunk oid 17 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1558: write (excl) oid 6 current snap is 32 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1558: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1558: writing smithi10548823-6 from 625293 to 1262511 tid 1 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1558: writing smithi10548823-6 from 1813763 to 2307866 tid 2 2024-04-20T02:52:30.070 INFO:tasks.rados.rados.0.smithi105.stdout:1558: writing smithi10548823-6 from 2978063 to 3025293 tid 3 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1554: done (7 left) 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1559: tier_promote oid 44 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1551: finishing set_chunk to oid 4 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 1086464 length: 1455104 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 0 (ObjNum 191 snap 19 seq_num 191) dirty exists 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1553: finishing rollback tid 1 to smithi10548823-10 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 10 v 0 (ObjNum 33 snap 0 seq_num 61363136) dirty dne 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1557: finishing set_chunk to oid 17 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1557: got expected ENOENT (src dne) 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1556: finishing rollback tid 0 to smithi10548823-35 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1556: finishing rollback tid 1 to smithi10548823-35 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1556: finishing rollback tid 2 to smithi10548823-35 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 66 (ObjNum 124 snap 9 seq_num 124) dirty exists 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1551: done (7 left) 2024-04-20T02:52:30.071 INFO:tasks.rados.rados.0.smithi105.stdout:1552: done (6 left) 2024-04-20T02:52:30.072 INFO:tasks.rados.rados.0.smithi105.stdout:1553: done (5 left) 2024-04-20T02:52:30.072 INFO:tasks.rados.rados.0.smithi105.stdout:1556: done (4 left) 2024-04-20T02:52:30.072 INFO:tasks.rados.rados.0.smithi105.stdout:1557: done (3 left) 2024-04-20T02:52:30.072 INFO:tasks.rados.rados.0.smithi105.stdout:1560: write oid 21 current snap is 32 2024-04-20T02:52:30.072 INFO:tasks.rados.rados.0.smithi105.stdout:1560: seq_num 300 ranges {642100=712467,1872245=637761} 2024-04-20T02:52:30.072 INFO:tasks.rados.rados.0.smithi105.stdout:1560: writing smithi10548823-21 from 642100 to 1354567 tid 1 2024-04-20T02:52:30.072 INFO:tasks.rados.rados.0.smithi105.stdout:1560: writing smithi10548823-21 from 1872245 to 2510006 tid 2 2024-04-20T02:52:30.072 INFO:tasks.rados.rados.0.smithi105.stdout:1561: read oid 18 snap -1 2024-04-20T02:52:30.072 INFO:tasks.rados.rados.0.smithi105.stdout:1561: expect (ObjNum 296 snap 32 seq_num 296) 2024-04-20T02:52:30.072 INFO:tasks.rados.rados.0.smithi105.stdout:1562: snap_remove snap 29 2024-04-20T02:52:30.072 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:30.059+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:30.161 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:30.160+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:30.179 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:52:30.200 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:30.200 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:30.200 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:30.200 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:30.210 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:30.262 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:30.262+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:30.292 INFO:tasks.rados.rados.0.smithi105.stdout:1562: done (5 left) 2024-04-20T02:52:30.292 INFO:tasks.rados.rados.0.smithi105.stdout:1563: read oid 17 snap -1 2024-04-20T02:52:30.292 INFO:tasks.rados.rados.0.smithi105.stdout:1563: expect deleted 2024-04-20T02:52:30.292 INFO:tasks.rados.rados.0.smithi105.stdout:1564: set_chunk oid 20 target oid 2024-04-20T02:52:30.303 INFO:tasks.rados.rados.0.smithi105.stdout:1564: set_chunk oid 20 offset: 2325504 length: 115712 target oid 84b64f81e3fb0ef8d8f74a13488d97d5b81e5f062c4ae835527a1b9a6ab77a99 offset: 0 snap -1 2024-04-20T02:52:30.304 INFO:tasks.rados.rados.0.smithi105.stdout:1565: tier_promote oid 41 2024-04-20T02:52:30.304 INFO:tasks.rados.rados.0.smithi105.stdout:1555: finishing write tid 1 to smithi10548823-13 2024-04-20T02:52:30.304 INFO:tasks.rados.rados.0.smithi105.stdout:1566: read oid 1 snap -1 2024-04-20T02:52:30.304 INFO:tasks.rados.rados.0.smithi105.stdout:1566: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:30.304 INFO:tasks.rados.rados.0.smithi105.stdout:1567: read oid 45 snap 28 2024-04-20T02:52:30.304 INFO:tasks.rados.rados.0.smithi105.stdout:1567: expect (ObjNum 220 snap 21 seq_num 220) 2024-04-20T02:52:30.304 INFO:tasks.rados.rados.0.smithi105.stdout:1568: set_chunk oid 37 target oid 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1568: set_chunk oid 37 offset: 805376 length: 308736 target oid e09a3e40a9a664a4b258f009ab328bd9ab0b9a770f3a4c34e9cb492a30686916 offset: 0 snap -1 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1569: read oid 15 snap -1 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1569: expect deleted 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1570: delete oid 3 current snap is 32 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1559: got (0) Success 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 0 (ObjNum 208 snap 19 seq_num 208) dirty exists 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1555: finishing write tid 2 to smithi10548823-13 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1558: finishing write tid 1 to smithi10548823-6 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1555: finishing write tid 3 to smithi10548823-13 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1555: finishing write tid 4 to smithi10548823-13 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1558: finishing write tid 2 to smithi10548823-6 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1558: finishing write tid 3 to smithi10548823-6 2024-04-20T02:52:30.319 INFO:tasks.rados.rados.0.smithi105.stdout:1558: finishing write tid 4 to smithi10548823-6 2024-04-20T02:52:30.320 INFO:tasks.rados.rados.0.smithi105.stdout:1558: finishing write tid 5 to smithi10548823-6 2024-04-20T02:52:30.320 INFO:tasks.rados.rados.0.smithi105.stdout:1558: oid 6 updating version 0 to 216 2024-04-20T02:52:30.320 INFO:tasks.rados.rados.0.smithi105.stdout:1558: oid 6 version 216 is already newer than 215 2024-04-20T02:52:30.320 INFO:tasks.rados.rados.0.smithi105.stdout:1558: oid 6 version 216 is already newer than 213 2024-04-20T02:52:30.320 INFO:tasks.rados.rados.0.smithi105.stdout:1558: oid 6 version 216 is already newer than 214 2024-04-20T02:52:30.320 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 216 (ObjNum 299 snap 32 seq_num 299) dirty exists 2024-04-20T02:52:30.320 INFO:tasks.rados.rados.0.smithi105.stdout:1558: left oid 6 (ObjNum 299 snap 32 seq_num 299) 2024-04-20T02:52:30.320 INFO:tasks.rados.rados.0.smithi105.stdout:1561: expect (ObjNum 296 snap 32 seq_num 296) 2024-04-20T02:52:30.362 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:30.362+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:30.390 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:30.390 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:30.390 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:30.390 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:30.390 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:30.400 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:30.450 INFO:tasks.rados.rados.0.smithi105.stdout:1555: finishing write tid 5 to smithi10548823-13 2024-04-20T02:52:30.450 INFO:tasks.rados.rados.0.smithi105.stdout:1555: oid 13 updating version 0 to 313 2024-04-20T02:52:30.450 INFO:tasks.rados.rados.0.smithi105.stdout:1555: oid 13 version 313 is already newer than 312 2024-04-20T02:52:30.450 INFO:tasks.rados.rados.0.smithi105.stdout:1555: oid 13 updating version 313 to 315 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1555: oid 13 version 315 is already newer than 314 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 315 (ObjNum 298 snap 32 seq_num 298) dirty exists 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1555: left oid 13 (ObjNum 298 snap 32 seq_num 298) 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1560: finishing write tid 1 to smithi10548823-21 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1560: finishing write tid 2 to smithi10548823-21 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1560: finishing write tid 3 to smithi10548823-21 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1560: finishing write tid 4 to smithi10548823-21 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1560: oid 21 updating version 0 to 145 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1560: oid 21 updating version 145 to 146 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1560: oid 21 updating version 146 to 147 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 147 (ObjNum 300 snap 32 seq_num 300) dirty exists 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1560: left oid 21 (ObjNum 300 snap 32 seq_num 300) 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1555: done (12 left) 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1558: done (11 left) 2024-04-20T02:52:30.451 INFO:tasks.rados.rados.0.smithi105.stdout:1559: done (10 left) 2024-04-20T02:52:30.452 INFO:tasks.rados.rados.0.smithi105.stdout:1560: done (9 left) 2024-04-20T02:52:30.452 INFO:tasks.rados.rados.0.smithi105.stdout:1561: done (8 left) 2024-04-20T02:52:30.452 INFO:tasks.rados.rados.0.smithi105.stdout:1570: done (7 left) 2024-04-20T02:52:30.452 INFO:tasks.rados.rados.0.smithi105.stdout:1571: read oid 48 snap -1 2024-04-20T02:52:30.452 INFO:tasks.rados.rados.0.smithi105.stdout:1571: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:30.452 INFO:tasks.rados.rados.0.smithi105.stdout:1572: tier_evict oid 31 2024-04-20T02:52:30.452 INFO:tasks.rados.rados.0.smithi105.stdout:1572: tier_evict oid 31 snap -1 2024-04-20T02:52:30.452 INFO:tasks.rados.rados.0.smithi105.stdout:1566: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:30.483 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:30.483+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:30.489 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:30.526 INFO:tasks.rados.rados.0.smithi105.stdout:1565: got (2) No such file or directory 2024-04-20T02:52:30.526 INFO:tasks.rados.rados.0.smithi105.stdout:1565: got expected ENOENT (src dne) 2024-04-20T02:52:30.527 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 212 (ObjNum 129 snap 0 seq_num 59787104) dirty dne 2024-04-20T02:52:30.527 INFO:tasks.rados.rados.0.smithi105.stdout:1564: finishing set_chunk to oid 20 2024-04-20T02:52:30.527 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 187 (ObjNum 289 snap 30 seq_num 289) dirty exists 2024-04-20T02:52:30.527 INFO:tasks.rados.rados.0.smithi105.stdout:1563: done (8 left) 2024-04-20T02:52:30.527 INFO:tasks.rados.rados.0.smithi105.stdout:1564: done (7 left) 2024-04-20T02:52:30.527 INFO:tasks.rados.rados.0.smithi105.stdout:1565: done (6 left) 2024-04-20T02:52:30.527 INFO:tasks.rados.rados.0.smithi105.stdout:1566: done (5 left) 2024-04-20T02:52:30.527 INFO:tasks.rados.rados.0.smithi105.stdout:1573: write oid 2 current snap is 32 2024-04-20T02:52:30.527 INFO:tasks.rados.rados.0.smithi105.stdout:1573: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-04-20T02:52:30.532 INFO:tasks.rados.rados.0.smithi105.stdout:1573: writing smithi10548823-2 from 675714 to 1138679 tid 1 2024-04-20T02:52:30.538 INFO:tasks.rados.rados.0.smithi105.stdout:1573: writing smithi10548823-2 from 1589209 to 2230639 tid 2 2024-04-20T02:52:30.543 INFO:tasks.rados.rados.0.smithi105.stdout:1573: writing smithi10548823-2 from 2943979 to 3075714 tid 3 2024-04-20T02:52:30.543 INFO:tasks.rados.rados.0.smithi105.stdout:1574: read oid 44 snap -1 2024-04-20T02:52:30.543 INFO:tasks.rados.rados.0.smithi105.stdout:1574: expect (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:52:30.543 INFO:tasks.rados.rados.0.smithi105.stdout:1575: snap_remove snap 26 2024-04-20T02:52:30.565 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:30.565+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:30.576 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:30.576 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:30.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:30.576 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:30.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.576 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:30.576 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:30.576 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 99.632180723999994, 2024-04-20T02:52:30.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:52:30.577 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.577 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.577 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.577 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.577 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.577 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:52:30.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.577 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:52:30.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:52:30.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:30.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:52:30.582 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.583 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.583 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 99.624702405999997, 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:52:30.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:52:30.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:52:30.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:52:30.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:52:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:52:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:52:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:52:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:52:30.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 99.619944507, 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:52:30.590 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:52:30.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:52:30.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:30.595 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 96.196515572999999, 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:52:30.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.602 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 96.196478690000006, 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:30.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:52:30.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:52:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:52:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:52:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:52:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:30.609 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 96.196454056999997, 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:30.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:52:30.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:52:30.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:52:30.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:52:30.615 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.615 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.615 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 96.189960646000003, 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.616 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:52:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:30.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:52:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:52:30.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 96.183896623999999, 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.624 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:30.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 96.173834858999996, 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.630 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:52:30.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:52:30.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:52:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:52:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:52:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 96.173142255000002, 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:30.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:52:30.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:52:30.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 96.173130600999997, 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:52:30.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:30.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.647 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.467066044999996, 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:52:30.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.448005328999997, 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.652 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:52:30.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:52:30.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:52:30.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:52:30.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.442059014999998, 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.658 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:52:30.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:52:30.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:52:30.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:52:30.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.441493574000006, 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.664 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:52:30.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.669 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.441467543000002, 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:52:30.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:52:30.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:52:30.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 34.439254742000003, 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.675 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:52:30.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.679 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 34.433905218, 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:52:30.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:52:30.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:52:30.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 34.433435261, 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 34.433412724999997, 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:30.689 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:52:30.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:30.693 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:30.695 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:30.695 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:30.695 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:30.695 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:30.695+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:30.696 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:30.697 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:30.795 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:30.796+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:30.865 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:30.865 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:30.865 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:30.865 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:30.875 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:30.912 INFO:teuthology.orchestra.run.smithi105.stdout:fsck success 2024-04-20T02:52:30.917 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:30.918 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:30.918+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:30.920 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-04-20T02:52:30.920 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-20T02:52:30.920 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-20T02:52:30.920 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-20T02:52:30.936 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:30.938 INFO:tasks.ceph.osd.2:Started 2024-04-20T02:52:30.938 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:30.958 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:30.959 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:30.959 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:30.959 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T02:52:30.959 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:30.959 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:30.959 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:30.959 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:30.959 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:30.960 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:31.002 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:31.002+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:31.004 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:31.004 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:31.014 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:31.061 INFO:teuthology.orchestra.run.smithi105.stderr:no valid command found; 10 closest matches: 2024-04-20T02:52:31.061 INFO:teuthology.orchestra.run.smithi105.stderr:0 2024-04-20T02:52:31.061 INFO:teuthology.orchestra.run.smithi105.stderr:1 2024-04-20T02:52:31.061 INFO:teuthology.orchestra.run.smithi105.stderr:2 2024-04-20T02:52:31.061 INFO:teuthology.orchestra.run.smithi105.stderr:abort 2024-04-20T02:52:31.061 INFO:teuthology.orchestra.run.smithi105.stderr:assert 2024-04-20T02:52:31.061 INFO:teuthology.orchestra.run.smithi105.stderr:config diff 2024-04-20T02:52:31.061 INFO:teuthology.orchestra.run.smithi105.stderr:config diff get 2024-04-20T02:52:31.061 INFO:teuthology.orchestra.run.smithi105.stderr:config get 2024-04-20T02:52:31.061 INFO:teuthology.orchestra.run.smithi105.stderr:config help [] 2024-04-20T02:52:31.062 INFO:teuthology.orchestra.run.smithi105.stderr:config set ... 2024-04-20T02:52:31.062 INFO:teuthology.orchestra.run.smithi105.stderr:admin_socket: invalid command 2024-04-20T02:52:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T02:52:31.063 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-20T02:52:31.105 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:31.105+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:31.184 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:31.184 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:31.194 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:31.238 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:31.238+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:31.294 INFO:tasks.rados.rados.0.smithi105.stdout:1575: done (7 left) 2024-04-20T02:52:31.294 INFO:tasks.rados.rados.0.smithi105.stdout:1576: rollback oid 38 current snap is 32 2024-04-20T02:52:31.294 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 38 to 28 2024-04-20T02:52:31.294 INFO:tasks.rados.rados.0.smithi105.stdout:1567: expect (ObjNum 220 snap 21 seq_num 220) 2024-04-20T02:52:31.298 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:52:31.302 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:31.301+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:31.328 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:31.328 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:31.338 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:31.423 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:31.422+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:31.468 INFO:tasks.rados.rados.0.smithi105.stdout:1568: finishing set_chunk to oid 37 2024-04-20T02:52:31.468 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 37 set_chunk returned error code -95 offset: 805376 length: 308736 2024-04-20T02:52:31.468 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 0 (ObjNum 263 snap 24 seq_num 263) dirty exists 2024-04-20T02:52:31.468 INFO:tasks.rados.rados.0.smithi105.stdout:1567: done (7 left) 2024-04-20T02:52:31.468 INFO:tasks.rados.rados.0.smithi105.stdout:1568: done (6 left) 2024-04-20T02:52:31.468 INFO:tasks.rados.rados.0.smithi105.stdout:1569: done (5 left) 2024-04-20T02:52:31.469 INFO:tasks.rados.rados.0.smithi105.stdout:1577: tier_promote oid 16 2024-04-20T02:52:31.469 INFO:tasks.rados.rados.0.smithi105.stdout:1572: got (22) Invalid argument 2024-04-20T02:52:31.469 INFO:tasks.rados.rados.0.smithi105.stdout:1571: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:31.502 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:31.501+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:31.509 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:31.509 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:31.520 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:31.527 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:31.527+0000 7f03997757c0 -1 Falling back to public interface 2024-04-20T02:52:31.603 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:31.603+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:31.676 INFO:tasks.rados.rados.0.smithi105.stdout:1574: expect (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:52:31.688 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:31.688 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:31.698 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:31.740 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:31.739+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:31.805 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:31.806+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:31.868 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:31.869 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:31.879 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:31.890 INFO:tasks.rados.rados.0.smithi105.stdout:1573: finishing write tid 1 to smithi10548823-2 2024-04-20T02:52:31.890 INFO:tasks.rados.rados.0.smithi105.stdout:1573: finishing write tid 2 to smithi10548823-2 2024-04-20T02:52:31.890 INFO:tasks.rados.rados.0.smithi105.stdout:1573: finishing write tid 3 to smithi10548823-2 2024-04-20T02:52:31.890 INFO:tasks.rados.rados.0.smithi105.stdout:1573: finishing write tid 4 to smithi10548823-2 2024-04-20T02:52:31.890 INFO:tasks.rados.rados.0.smithi105.stdout:1573: finishing write tid 5 to smithi10548823-2 2024-04-20T02:52:31.890 INFO:tasks.rados.rados.0.smithi105.stdout:1573: oid 2 updating version 0 to 201 2024-04-20T02:52:31.890 INFO:tasks.rados.rados.0.smithi105.stdout:1573: oid 2 updating version 201 to 203 2024-04-20T02:52:31.890 INFO:tasks.rados.rados.0.smithi105.stdout:1573: oid 2 version 203 is already newer than 200 2024-04-20T02:52:31.890 INFO:tasks.rados.rados.0.smithi105.stdout:1573: oid 2 version 203 is already newer than 202 2024-04-20T02:52:31.890 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 203 (ObjNum 302 snap 32 seq_num 302) dirty exists 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1573: left oid 2 (ObjNum 302 snap 32 seq_num 302) 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1576: finishing rollback tid 1 to smithi10548823-38 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 38 v 0 (ObjNum 97 snap 0 seq_num 58976432) dirty dne 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1577: got (2) No such file or directory 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1577: got expected ENOENT (src dne) 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 179 (ObjNum 305 snap 0 seq_num 1) dirty dne 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1571: done (5 left) 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1572: done (4 left) 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1573: done (3 left) 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1574: done (2 left) 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1576: done (1 left) 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1577: done (0 left) 2024-04-20T02:52:31.891 INFO:tasks.rados.rados.0.smithi105.stdout:1578: set_chunk oid 14 target oid 2024-04-20T02:52:31.921 INFO:tasks.rados.rados.0.smithi105.stdout:1578: set_chunk oid 14 offset: 1043968 length: 1342464 target oid 798ff6f08cf3bdaa5002c8cbe14cd0ee5da3adda2461e54fb5e25bc12807bf29 offset: 0 snap -1 2024-04-20T02:52:31.921 INFO:tasks.rados.rados.0.smithi105.stdout:1579: set_chunk oid 42 target oid 2024-04-20T02:52:31.921 INFO:tasks.rados.rados.0.smithi105.stdout:1579: set_chunk oid 42 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:31.921 INFO:tasks.rados.rados.0.smithi105.stdout:1580: write (excl) oid 24 current snap is 32 2024-04-20T02:52:31.921 INFO:tasks.rados.rados.0.smithi105.stdout:1580: seq_num 303 ranges {692521=538214,1931338=785088} 2024-04-20T02:52:31.922 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:31.922+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:31.927 INFO:tasks.rados.rados.0.smithi105.stdout:1580: writing smithi10548823-24 from 692521 to 1230735 tid 1 2024-04-20T02:52:31.936 INFO:tasks.rados.rados.0.smithi105.stdout:1580: writing smithi10548823-24 from 1931338 to 2716426 tid 2 2024-04-20T02:52:31.936 INFO:tasks.rados.rados.0.smithi105.stdout:1581: read oid 36 snap -1 2024-04-20T02:52:31.936 INFO:tasks.rados.rados.0.smithi105.stdout:1581: expect (ObjNum 244 snap 22 seq_num 244) 2024-04-20T02:52:31.936 INFO:tasks.rados.rados.0.smithi105.stdout:1582: snap_create 2024-04-20T02:52:31.936 INFO:tasks.rados.rados.0.smithi105.stdout:1579: finishing set_chunk to oid 42 2024-04-20T02:52:31.936 INFO:tasks.rados.rados.0.smithi105.stdout:1579: got expected ERANGE (src dne) 2024-04-20T02:52:31.936 INFO:tasks.rados.rados.0.smithi105.stdout:1578: finishing set_chunk to oid 14 2024-04-20T02:52:31.936 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 107 (ObjNum 274 snap 26 seq_num 274) dirty exists 2024-04-20T02:52:31.958 INFO:tasks.rados.rados.0.smithi105.stdout:1580: finishing write tid 1 to smithi10548823-24 2024-04-20T02:52:31.963 INFO:tasks.rados.rados.0.smithi105.stdout:1580: finishing write tid 2 to smithi10548823-24 2024-04-20T02:52:31.963 INFO:tasks.rados.rados.0.smithi105.stdout:1580: finishing write tid 3 to smithi10548823-24 2024-04-20T02:52:31.963 INFO:tasks.rados.rados.0.smithi105.stdout:1580: finishing write tid 4 to smithi10548823-24 2024-04-20T02:52:31.963 INFO:tasks.rados.rados.0.smithi105.stdout:1580: oid 24 updating version 0 to 333 2024-04-20T02:52:31.963 INFO:tasks.rados.rados.0.smithi105.stdout:1580: oid 24 version 333 is already newer than 331 2024-04-20T02:52:31.964 INFO:tasks.rados.rados.0.smithi105.stdout:1580: oid 24 version 333 is already newer than 332 2024-04-20T02:52:31.964 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 333 (ObjNum 303 snap 32 seq_num 303) dirty exists 2024-04-20T02:52:31.964 INFO:tasks.rados.rados.0.smithi105.stdout:1580: left oid 24 (ObjNum 303 snap 32 seq_num 303) 2024-04-20T02:52:31.964 INFO:tasks.rados.rados.0.smithi105.stdout:1581: expect (ObjNum 244 snap 22 seq_num 244) 2024-04-20T02:52:32.007 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:32.008+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:32.013 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:32.013 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:32.024 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:32.108 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:32.109+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:32.207 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:32.207 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:32.209 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:32.208+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:32.218 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:32.307 INFO:tasks.rados.rados.0.smithi105.stdout:1578: done (4 left) 2024-04-20T02:52:32.307 INFO:tasks.rados.rados.0.smithi105.stdout:1579: done (3 left) 2024-04-20T02:52:32.307 INFO:tasks.rados.rados.0.smithi105.stdout:1580: done (2 left) 2024-04-20T02:52:32.307 INFO:tasks.rados.rados.0.smithi105.stdout:1581: done (1 left) 2024-04-20T02:52:32.307 INFO:tasks.rados.rados.0.smithi105.stdout:1582: done (0 left) 2024-04-20T02:52:32.308 INFO:tasks.rados.rados.0.smithi105.stdout:1583: read oid 18 snap -1 2024-04-20T02:52:32.308 INFO:tasks.rados.rados.0.smithi105.stdout:1583: expect (ObjNum 296 snap 32 seq_num 296) 2024-04-20T02:52:32.308 INFO:tasks.rados.rados.0.smithi105.stdout:1584: read oid 48 snap 32 2024-04-20T02:52:32.308 INFO:tasks.rados.rados.0.smithi105.stdout:1584: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:32.308 INFO:tasks.rados.rados.0.smithi105.stdout:1585: write (excl) oid 36 current snap is 33 2024-04-20T02:52:32.308 INFO:tasks.rados.rados.0.smithi105.stdout:1585: seq_num 304 ranges {709328=613463,1989820=645099} 2024-04-20T02:52:32.309 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:32.308+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:32.316 INFO:tasks.rados.rados.0.smithi105.stdout:1585: writing smithi10548823-36 from 709328 to 1322791 tid 1 2024-04-20T02:52:32.324 INFO:tasks.rados.rados.0.smithi105.stdout:1585: writing smithi10548823-36 from 1989820 to 2634919 tid 2 2024-04-20T02:52:32.324 INFO:tasks.rados.rados.0.smithi105.stdout:1586: write (excl) oid 31 current snap is 33 2024-04-20T02:52:32.324 INFO:tasks.rados.rados.0.smithi105.stdout:1586: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-04-20T02:52:32.331 INFO:tasks.rados.rados.0.smithi105.stdout:1586: writing smithi10548823-31 from 726135 to 1131200 tid 1 2024-04-20T02:52:32.341 INFO:tasks.rados.rados.0.smithi105.stdout:1586: writing smithi10548823-31 from 1764655 to 2553412 tid 2 2024-04-20T02:52:32.345 INFO:tasks.rados.rados.0.smithi105.stdout:1586: writing smithi10548823-31 from 3026248 to 3126135 tid 3 2024-04-20T02:52:32.345 INFO:tasks.rados.rados.0.smithi105.stdout:1587: write oid 24 current snap is 33 2024-04-20T02:52:32.346 INFO:tasks.rados.rados.0.smithi105.stdout:1587: seq_num 306 ranges {742942=480314,1823137=648768} 2024-04-20T02:52:32.351 INFO:tasks.rados.rados.0.smithi105.stdout:1587: writing smithi10548823-24 from 742942 to 1223256 tid 1 2024-04-20T02:52:32.360 INFO:tasks.rados.rados.0.smithi105.stdout:1587: writing smithi10548823-24 from 1823137 to 2471905 tid 2 2024-04-20T02:52:32.360 INFO:tasks.rados.rados.0.smithi105.stdout:1588: tier_promote oid 27 2024-04-20T02:52:32.360 INFO:tasks.rados.rados.0.smithi105.stdout:1589: write (excl) oid 2 current snap is 33 2024-04-20T02:52:32.360 INFO:tasks.rados.rados.0.smithi105.stdout:1589: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-04-20T02:52:32.368 INFO:tasks.rados.rados.0.smithi105.stdout:1589: writing smithi10548823-2 from 759749 to 1315312 tid 1 2024-04-20T02:52:32.377 INFO:tasks.rados.rados.0.smithi105.stdout:1589: writing smithi10548823-2 from 1765266 to 2557692 tid 2 2024-04-20T02:52:32.379 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:32.380 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:32.381 INFO:tasks.rados.rados.0.smithi105.stdout:1589: writing smithi10548823-2 from 2964741 to 3159749 tid 3 2024-04-20T02:52:32.381 INFO:tasks.rados.rados.0.smithi105.stdout:1590: read oid 16 snap -1 2024-04-20T02:52:32.381 INFO:tasks.rados.rados.0.smithi105.stdout:1590: expect deleted 2024-04-20T02:52:32.381 INFO:tasks.rados.rados.0.smithi105.stdout:1591: read oid 22 snap 32 2024-04-20T02:52:32.381 INFO:tasks.rados.rados.0.smithi105.stdout:1591: expect (ObjNum 292 snap 32 seq_num 292) 2024-04-20T02:52:32.382 INFO:tasks.rados.rados.0.smithi105.stdout:1592: read oid 8 snap 30 2024-04-20T02:52:32.382 INFO:tasks.rados.rados.0.smithi105.stdout:1592: expect (ObjNum 257 snap 22 seq_num 257) 2024-04-20T02:52:32.382 INFO:tasks.rados.rados.0.smithi105.stdout:1593: read oid 33 snap -1 2024-04-20T02:52:32.382 INFO:tasks.rados.rados.0.smithi105.stdout:1593: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:52:32.382 INFO:tasks.rados.rados.0.smithi105.stdout:1594: set_chunk oid 6 target oid 2024-04-20T02:52:32.399 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:32.410 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:32.410+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:32.427 INFO:tasks.rados.rados.0.smithi105.stdout:1594: set_chunk oid 6 offset: 384512 length: 1266176 target oid bfc7dd417bcf3f01b5a2291b1e195c3f2a593a82dcb240fa2468118384dba516 offset: 0 snap -1 2024-04-20T02:52:32.427 INFO:tasks.rados.rados.0.smithi105.stdout:1595: snap_remove snap 31 2024-04-20T02:52:32.512 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:32.511+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:32.529 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:32.529 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:32.540 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:32.613 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:32.613+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:32.708 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:32.708 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:32.713 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:32.714+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:32.718 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:32.815 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:32.815+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:32.846 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:32.846 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:32.857 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:32.915 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:32.915+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:32.961 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:32.987 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:32.988 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:32.999 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:33.040 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:33.040+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:33.109 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:33.117 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:33.117+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:33.129 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:33.150 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:33.170 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:33.175 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 106.589039619, 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.176 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.177 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:52:33.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.177 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:33.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:982 4.7 4:e9876a63:::smithi10548823-30:head [write 1798066~477679] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:52:33.182 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 106.573021341, 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050505823999999998, 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 982 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601159+0000", 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601157+0000", 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601272+0000", 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114831 2024-04-20T02:52:33.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601288+0000", 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6311999999999999e-05 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601301+0000", 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2734e-05 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607779+0000", 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0064778869999999999 2024-04-20T02:52:33.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607842+0000", 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3867000000000004e-05 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607941+0000", 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8245999999999994e-05 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635299+0000", 2024-04-20T02:52:33.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027358352999999998 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635314+0000", 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4803999999999999e-05 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651583+0000", 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016269084 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651626+0000", 2024-04-20T02:52:33.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2580000000000002e-05 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651665+0000", 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8940999999999999e-05 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:983 4.7 4:e9876a63:::smithi10548823-30:head [write 2742758~457217] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 106.57286811199999, 2024-04-20T02:52:33.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050453916000000001, 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 983 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601312+0000", 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601311+0000", 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601401+0000", 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9786999999999999e-05 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601409+0000", 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601416+0000", 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0319999999999998e-06 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624685+0000", 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02326831 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624793+0000", 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108553 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.624976+0000", 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000183248 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635942+0000", 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010965775000000001 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651361+0000", 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015418704 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651697+0000", 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033656300000000001 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651732+0000", 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5052999999999998e-05 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651766+0000", 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3361000000000001e-05 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:984 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (56) in=7b,truncate 3310946] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 106.572758103, 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050457779000000001, 2024-04-20T02:52:33.194 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 984 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601422+0000", 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601421+0000", 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601425+0000", 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3579999999999999e-06 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601431+0000", 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0680000000000002e-06 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601446+0000", 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708e-05 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635347+0000", 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.033901453999999998 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635449+0000", 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101945 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635607+0000", 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157799 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651428+0000", 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015821293 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651433+0000", 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6829999999999999e-06 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651800+0000", 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036674400000000001 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651834+0000", 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4748e-05 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651880+0000", 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5408000000000003e-05 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:985 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 106.572727727, 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050647632999999997, 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 985 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601452+0000", 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601455+0000", 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7539999999999998e-06 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601462+0000", 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9940000000000003e-06 2024-04-20T02:52:33.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.601467+0000", 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2469999999999999e-06 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635846+0000", 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034379276 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635895+0000", 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.935e-05 2024-04-20T02:52:33.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651909+0000", 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016013461 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651986+0000", 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7125000000000004e-05 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.652100+0000", 2024-04-20T02:52:33.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113892 2024-04-20T02:52:33.205 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.205 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.205 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.206 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.206 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:33.206 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 106.56831030799999, 2024-04-20T02:52:33.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:52:33.206 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.206 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:52:33.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:33.212 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 106.559058557, 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:52:33.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:52:33.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:52:33.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:52:33.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:988 4.f 4:f72cd604:::smithi10548823-29:head [write 3286534~41219] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:52:33.218 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 106.55653641399999, 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061042875000000003, 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 988 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617644+0000", 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617642+0000", 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617685+0000", 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2843000000000003e-05 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617711+0000", 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5537999999999998e-05 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617721+0000", 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0030999999999999e-05 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667036+0000", 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049315091999999998 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667132+0000", 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6148000000000002e-05 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667288+0000", 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015619000000000001 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668850+0000", 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015623849999999999 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.670783+0000", 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001932787 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678611+0000", 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0078280239999999994 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678652+0000", 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0417000000000003e-05 2024-04-20T02:52:33.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678686+0000", 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4833999999999999e-05 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:989 4.f 4:f72cd604:::smithi10548823-29:head [setxattr _header (56) in=7b,truncate 3327753] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 106.55645154699999, 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061497643999999997, 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.224 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 989 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:33.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617728+0000", 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617732+0000", 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3050000000000003e-06 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617737+0000", 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8659999999999997e-06 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617742+0000", 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1540000000000001e-06 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668397+0000", 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050654883999999997 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668448+0000", 2024-04-20T02:52:33.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0408000000000003e-05 2024-04-20T02:52:33.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668529+0000", 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1583999999999997e-05 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.669088+0000", 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055875100000000002 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.673433+0000", 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043453329999999998 2024-04-20T02:52:33.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679132+0000", 2024-04-20T02:52:33.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0056992830000000003 2024-04-20T02:52:33.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679178+0000", 2024-04-20T02:52:33.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5945999999999997e-05 2024-04-20T02:52:33.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679226+0000", 2024-04-20T02:52:33.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7709e-05 2024-04-20T02:52:33.231 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.231 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:990 4.f 4:f72cd604:::smithi10548823-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 106.55643393699999, 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061721755000000003, 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.232 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 990 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617746+0000", 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617748+0000", 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2850000000000001e-06 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617756+0000", 2024-04-20T02:52:33.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1249999999999993e-06 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.617761+0000", 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1059999999999999e-06 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668720+0000", 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050958677000000001 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:33.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668814+0000", 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4201000000000002e-05 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679271+0000", 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01045719 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679359+0000", 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7811000000000003e-05 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.679468+0000", 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010873400000000001 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1144 4.d e35/22 4:b376c20e:::smithi10548823-49:head v 35'29, mlcod=34'26)", 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 102.207067329, 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058137664999999998, 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.237 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967113+0000", 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.967112+0000", 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981895+0000", 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014783522 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981906+0000", 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0560000000000001e-05 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981912+0000", 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981956+0000", 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3455000000000003e-05 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.981979+0000", 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3475000000000001e-05 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025222+0000", 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043242622000000001 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.025250+0000", 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8520000000000001e-05 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.241 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.762059340999997, 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:33.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:52:33.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:52:33.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.756054552999998, 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:52:33.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:52:33.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:52:33.249 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2279 4.d e76/22 4:b9e59c86:::smithi10548823-2:head v 76'104, mlcod=73'100)", 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 59.032838429999998, 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056137746000000002, 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141342+0000", 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.141340+0000", 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142751+0000", 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014113540000000001 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142770+0000", 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8450000000000001e-05 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142780+0000", 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0284e-05 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.142982+0000", 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000202244 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.143007+0000", 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4929000000000001e-05 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197446+0000", 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054438519999999997 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.197479+0000", 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3597999999999999e-05 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2490 4.9 e90/81 4:9c87d36d:::smithi10548823-24:head v 90'246, mlcod=88'242)", 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 45.929739507000001, 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055338261999999999, 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244441+0000", 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.244440+0000", 2024-04-20T02:52:33.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245684+0000", 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001243755 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245696+0000", 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.171e-05 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245706+0000", 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0236e-05 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245756+0000", 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0571000000000001e-05 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.245784+0000", 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7841999999999999e-05 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.299737+0000", 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053952420000000001 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:47.299779+0000", 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2014000000000001e-05 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.258 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2654 4.7 4:e9876a63:::smithi10548823-30:head [write 2075401~451067] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.703306335999997, 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063177293999999995, 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2654 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470874+0000", 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.470871+0000", 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471109+0000", 2024-04-20T02:52:33.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023773800000000001 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471129+0000", 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0100999999999999e-05 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.471139+0000", 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0427e-05 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.493743+0000", 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02260334 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.493897+0000", 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015392800000000001 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.494161+0000", 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026414899999999998 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510453+0000", 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016292054 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510509+0000", 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6709000000000002e-05 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533920+0000", 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023410713999999999 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533989+0000", 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8421000000000001e-05 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534051+0000", 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2398999999999997e-05 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2655 4.7 4:e9876a63:::smithi10548823-30:head [write 3107008~648769] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:52:33.264 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.695403304999999, 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055363825999999998, 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2655 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478777+0000", 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478775+0000", 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478959+0000", 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018453600000000001 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478981+0000", 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1523e-05 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.478991+0000", 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0288e-05 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510544+0000", 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031552795000000002 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510667+0000", 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012335699999999999 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.510928+0000", 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000260487 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533864+0000", 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022936926 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533908+0000", 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3498000000000003e-05 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534060+0000", 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015223999999999999 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534113+0000", 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2612999999999999e-05 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534141+0000", 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7696e-05 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2656 4.7 4:e9876a63:::smithi10548823-30:head [setxattr _header (57) in=7b,truncate 3798382] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.695173228000002, 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055660077000000002, 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.270 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2656 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479007+0000", 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479006+0000", 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479011+0000", 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.578e-06 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479015+0000", 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3869999999999997e-06 2024-04-20T02:52:33.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479019+0000", 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.703e-06 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533236+0000", 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054217551000000003 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533355+0000", 2024-04-20T02:52:33.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011906899999999999 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,5", 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.533595+0000", 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023979599999999999 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534575+0000", 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00097944500000000001 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534599+0000", 2024-04-20T02:52:33.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4371000000000001e-05 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534605+0000", 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534631+0000", 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6469999999999999e-05 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534667+0000", 2024-04-20T02:52:33.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5822999999999999e-05 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2657 4.7 4:e9876a63:::smithi10548823-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.695157928999997, 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055802286, 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.276 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2657 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479022+0000", 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479024+0000", 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.779e-06 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479027+0000", 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4529999999999998e-06 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479030+0000", 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.993e-06 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534184+0000", 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055154305000000001 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534295+0000", 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110665 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534694+0000", 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039911500000000002 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534753+0000", 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8649e-05 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534824+0000", 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1589999999999997e-05 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2658 4.7 4:e2c0d65b:::smithi10548823-46:head [read 561152~397312] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.695147728000002, 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055503328999999997, 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2658 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479032+0000", 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479034+0000", 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5600000000000001e-06 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479036+0000", 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6280000000000001e-06 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.479039+0000", 2024-04-20T02:52:33.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5969999999999999e-06 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534331+0000", 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055292556 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534409+0000", 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7411999999999997e-05 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:51.534536+0000", 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012682500000000001 2024-04-20T02:52:33.284 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.285 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.285 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.285 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.285 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.285 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:33.287 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:33.287 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:33.287 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:33.287 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:33.288 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:33.288 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T02:52:33.288 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:33.289 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:33.289 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:33.289 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:33.289 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:33.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:33.289 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:33.286+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:33.307 INFO:tasks.rados.rados.0.smithi105.stdout:1595: done (12 left) 2024-04-20T02:52:33.308 INFO:tasks.rados.rados.0.smithi105.stdout:1596: tier_promote oid 13 2024-04-20T02:52:33.308 INFO:tasks.rados.rados.0.smithi105.stdout:1584: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:33.311 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:52:33.387 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:33.387+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:33.452 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:33.453 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:33.453 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:33.453 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:33.467 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:33.488 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:33.489+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:33.533 INFO:tasks.rados.rados.0.smithi105.stdout:1583: expect (ObjNum 296 snap 32 seq_num 296) 2024-04-20T02:52:33.589 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:33.589+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:33.596 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:33.596 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:33.596 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:33.596 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:33.596 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:33.606 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:33.665 INFO:tasks.rados.rados.0.smithi105.stdout:1585: finishing write tid 1 to smithi10548823-36 2024-04-20T02:52:33.665 INFO:tasks.rados.rados.0.smithi105.stdout:1583: done (12 left) 2024-04-20T02:52:33.665 INFO:tasks.rados.rados.0.smithi105.stdout:1584: done (11 left) 2024-04-20T02:52:33.665 INFO:tasks.rados.rados.0.smithi105.stdout:1590: done (10 left) 2024-04-20T02:52:33.665 INFO:tasks.rados.rados.0.smithi105.stdout:1597: set_chunk oid 16 target oid 2024-04-20T02:52:33.666 INFO:tasks.rados.rados.0.smithi105.stdout:1597: set_chunk oid 16 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:33.666 INFO:tasks.rados.rados.0.smithi105.stdout:1598: read oid 42 snap 30 2024-04-20T02:52:33.666 INFO:tasks.rados.rados.0.smithi105.stdout:1598: expect deleted 2024-04-20T02:52:33.666 INFO:tasks.rados.rados.0.smithi105.stdout:1599: rollback oid 40 current snap is 33 2024-04-20T02:52:33.666 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 40 to 28 2024-04-20T02:52:33.666 INFO:tasks.rados.rados.0.smithi105.stdout:1593: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:52:33.689 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:33.690+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:33.790 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:33.790+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 107.272178631, 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.817 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:52:33.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:52:33.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:52:33.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:52:33.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:52:33.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 107.26595755699999, 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.828 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.830 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.831 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 107.257557353, 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:33.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:52:33.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:52:33.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:33.837 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 107.25721470800001, 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:52:33.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:52:33.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:52:33.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:33.842 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 99.450892705000001, 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:33.848 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 99.443629743000002, 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:33.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:33.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:52:33.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.851 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:52:33.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 99.437687213999993, 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.855 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:52:33.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:52:33.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:52:33.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:52:33.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 99.437484221999995, 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.861 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.867 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 99.437461295000006, 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.868 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:52:33.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:52:33.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 99.427377561, 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:52:33.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:52:33.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 99.410492153000007, 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:33.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:52:33.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:52:33.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 99.394215712000005, 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:52:33.881 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:52:33.885 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 84.369660240000002, 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:33.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:52:33.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:52:33.891 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.055344464000001, 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:33.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.897 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2545 4.9 e91/81 4:95e19a88:::smithi10548823-40:head v 91'253, mlcod=91'251)", 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 45.400707349000001, 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054757404000000003, 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:33.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415210+0000", 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416004+0000", 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079381199999999997 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416022+0000", 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7969e-05 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416031+0000", 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9279999999999999e-06 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416100+0000", 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8891000000000002e-05 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416128+0000", 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8203000000000001e-05 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469912+0000", 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053783794000000003 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469968+0000", 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6474000000000002e-05 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.901 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.467678542000002, 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:52:33.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:52:33.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.907 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2700 4.0 4:007b2a87:::smithi10548823-39:head [write 1967200~714747] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.459188347999998, 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061846287, 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2700 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356728+0000", 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356879+0000", 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000150958 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356894+0000", 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5132e-05 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356906+0000", 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2071e-05 2024-04-20T02:52:33.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366666+0000", 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0097595289999999994 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366774+0000", 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108527 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366956+0000", 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181359 2024-04-20T02:52:33.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390515+0000", 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023559902000000001 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406790+0000", 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016274555999999999 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418488+0000", 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011697591 2024-04-20T02:52:33.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418529+0000", 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1545000000000003e-05 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418576+0000", 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6903000000000001e-05 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.913 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2701 4.0 4:007b2a87:::smithi10548823-39:head [write 3421983~426820] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.452012355999997, 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054799176999999998, 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2701 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363904+0000", 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364154+0000", 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000250179 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364171+0000", 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6748e-05 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364182+0000", 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1371e-05 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390552+0000", 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026369607 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390652+0000", 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010059499999999999 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390857+0000", 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020445199999999999 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418224+0000", 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027367316999999999 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418282+0000", 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8279000000000002e-05 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418618+0000", 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000335967 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418661+0000", 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2614999999999998e-05 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418705+0000", 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3993999999999999e-05 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2702 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (57) in=7b,truncate 3848803] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:33.919 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.451726301999997, 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058669165000000002, 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2702 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364191+0000", 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364196+0000", 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4490000000000001e-06 2024-04-20T02:52:33.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364201+0000", 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5040000000000002e-06 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364206+0000", 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.405895+0000", 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041689047 2024-04-20T02:52:33.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406021+0000", 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000125561 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406541+0000", 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052000699999999996 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418451+0000", 2024-04-20T02:52:33.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011910041999999999 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418747+0000", 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029622399999999999 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422704+0000", 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039571859999999997 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422781+0000", 2024-04-20T02:52:33.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.661e-05 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422861+0000", 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9987999999999994e-05 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2703 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 41.451707249999998, 2024-04-20T02:52:33.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058940278999999998, 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2703 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364213+0000", 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4830000000000002e-06 2024-04-20T02:52:33.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364220+0000", 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.007e-06 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364225+0000", 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0350000000000001e-06 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406820+0000", 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042595135999999999 2024-04-20T02:52:33.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406923+0000", 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102846 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422897+0000", 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015974202999999999 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423015+0000", 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117581 2024-04-20T02:52:33.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:33.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:33.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:33.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423151+0000", 2024-04-20T02:52:33.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013635899999999999 2024-04-20T02:52:33.930 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.930 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.930 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.930 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:33.930 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:33.930 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:33.932 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:33.932 INFO:tasks.rados.rados.0.smithi105.stdout:1586: finishing write tid 1 to smithi10548823-31 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1585: finishing write tid 2 to smithi10548823-36 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1585: finishing write tid 3 to smithi10548823-36 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1585: finishing write tid 4 to smithi10548823-36 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1585: oid 36 updating version 0 to 336 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1585: oid 36 version 336 is already newer than 335 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1585: oid 36 updating version 336 to 337 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 337 (ObjNum 304 snap 33 seq_num 304) dirty exists 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1585: left oid 36 (ObjNum 304 snap 33 seq_num 304) 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1587: finishing write tid 1 to smithi10548823-24 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1585: done (12 left) 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1593: done (11 left) 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1600: tier_flush oid 30 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1600: tier_flush oid 30 snap -1 2024-04-20T02:52:33.933 INFO:tasks.rados.rados.0.smithi105.stdout:1592: expect (ObjNum 257 snap 22 seq_num 257) 2024-04-20T02:52:33.934 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:33.931+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:33.977 INFO:tasks.rados.rados.0.smithi105.stdout:1588: got (0) Success 2024-04-20T02:52:33.977 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 152 (ObjNum 247 snap 22 seq_num 247) dirty exists 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1587: finishing write tid 2 to smithi10548823-24 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1587: finishing write tid 3 to smithi10548823-24 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1588: done (11 left) 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1592: done (10 left) 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1601: read oid 27 snap -1 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1601: expect (ObjNum 247 snap 22 seq_num 247) 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1602: set_chunk oid 9 target oid 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1602: set_chunk oid 9 offset: 1721856 length: 55808 target oid smithi10548823- offset: 1721856 snap 30 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1603: read oid 26 snap -1 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1603: expect (ObjNum 223 snap 22 seq_num 223) 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1604: read oid 20 snap 32 2024-04-20T02:52:33.978 INFO:tasks.rados.rados.0.smithi105.stdout:1604: expect (ObjNum 289 snap 30 seq_num 289) 2024-04-20T02:52:33.979 INFO:tasks.rados.rados.0.smithi105.stdout:1605: read oid 4 snap -1 2024-04-20T02:52:33.979 INFO:tasks.rados.rados.0.smithi105.stdout:1605: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:33.979 INFO:tasks.rados.rados.0.smithi105.stdout:1606: read oid 12 snap -1 2024-04-20T02:52:33.979 INFO:tasks.rados.rados.0.smithi105.stdout:1606: expect deleted 2024-04-20T02:52:33.979 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:33.979 INFO:tasks.rados.rados.0.smithi105.stdout:1587: finishing write tid 4 to smithi10548823-24 2024-04-20T02:52:33.979 INFO:tasks.rados.rados.0.smithi105.stdout:1587: oid 24 updating version 0 to 340 2024-04-20T02:52:33.980 INFO:tasks.rados.rados.0.smithi105.stdout:1587: oid 24 version 340 is already newer than 339 2024-04-20T02:52:33.980 INFO:tasks.rados.rados.0.smithi105.stdout:1587: oid 24 updating version 340 to 341 2024-04-20T02:52:33.980 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 341 (ObjNum 306 snap 33 seq_num 306) dirty exists 2024-04-20T02:52:33.980 INFO:tasks.rados.rados.0.smithi105.stdout:1587: left oid 24 (ObjNum 306 snap 33 seq_num 306) 2024-04-20T02:52:33.980 INFO:tasks.rados.rados.0.smithi105.stdout:1591: expect (ObjNum 292 snap 32 seq_num 292) 2024-04-20T02:52:34.032 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:34.032+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:34.070 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:34.070 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:34.070 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:34.070 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:34.080 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:34.133 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:34.133+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:34.203 INFO:tasks.rados.rados.0.smithi105.stdout:1586: finishing write tid 2 to smithi10548823-31 2024-04-20T02:52:34.203 INFO:tasks.rados.rados.0.smithi105.stdout:1586: finishing write tid 3 to smithi10548823-31 2024-04-20T02:52:34.203 INFO:tasks.rados.rados.0.smithi105.stdout:1586: finishing write tid 4 to smithi10548823-31 2024-04-20T02:52:34.203 INFO:tasks.rados.rados.0.smithi105.stdout:1586: finishing write tid 5 to smithi10548823-31 2024-04-20T02:52:34.203 INFO:tasks.rados.rados.0.smithi105.stdout:1586: oid 31 updating version 0 to 206 2024-04-20T02:52:34.203 INFO:tasks.rados.rados.0.smithi105.stdout:1586: oid 31 updating version 206 to 208 2024-04-20T02:52:34.203 INFO:tasks.rados.rados.0.smithi105.stdout:1586: oid 31 version 208 is already newer than 207 2024-04-20T02:52:34.203 INFO:tasks.rados.rados.0.smithi105.stdout:1586: oid 31 version 208 is already newer than 205 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 208 (ObjNum 305 snap 33 seq_num 305) dirty exists 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1586: left oid 31 (ObjNum 305 snap 33 seq_num 305) 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1589: finishing write tid 1 to smithi10548823-2 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1586: done (15 left) 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1587: done (14 left) 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1591: done (13 left) 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1607: read oid 46 snap -1 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1607: expect (ObjNum 232 snap 22 seq_num 232) 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1608: tier_evict oid 19 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1608: tier_evict oid 19 snap -1 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1589: finishing write tid 2 to smithi10548823-2 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1589: finishing write tid 3 to smithi10548823-2 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1589: finishing write tid 4 to smithi10548823-2 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1589: finishing write tid 5 to smithi10548823-2 2024-04-20T02:52:34.204 INFO:tasks.rados.rados.0.smithi105.stdout:1589: oid 2 updating version 0 to 211 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:1589: oid 2 version 211 is already newer than 210 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:1589: oid 2 updating version 211 to 213 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:1589: oid 2 version 213 is already newer than 212 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 213 (ObjNum 307 snap 33 seq_num 307) dirty exists 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:1589: left oid 2 (ObjNum 307 snap 33 seq_num 307) 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:1594: finishing set_chunk to oid 6 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 216 (ObjNum 299 snap 32 seq_num 299) dirty exists 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:1596: got (0) Success 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 315 (ObjNum 298 snap 32 seq_num 298) dirty exists 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:1589: done (14 left) 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:1594: done (13 left) 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:1596: done (12 left) 2024-04-20T02:52:34.205 INFO:tasks.rados.rados.0.smithi105.stdout:1609: set_chunk oid 43 target oid 2024-04-20T02:52:34.234 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:34.234+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:34.251 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:34.251 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:34.251 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:34.251 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:34.251 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:34.261 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:34.311 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:52:34.335 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:34.334+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:34.365 INFO:tasks.rados.rados.0.smithi105.stdout:1609: set_chunk oid 43 offset: 145408 length: 2260480 target oid dbab0d4d95a93d18beb464c288a6e288ef32dc08f865a82956718b6980594a3e offset: 0 snap -1 2024-04-20T02:52:34.365 INFO:tasks.rados.rados.0.smithi105.stdout:1610: read oid 3 snap -1 2024-04-20T02:52:34.365 INFO:tasks.rados.rados.0.smithi105.stdout:1610: expect deleted 2024-04-20T02:52:34.365 INFO:tasks.rados.rados.0.smithi105.stdout:1611: set_chunk oid 5 target oid 2024-04-20T02:52:34.396 INFO:tasks.rados.rados.0.smithi105.stdout:1611: set_chunk oid 5 offset: 423936 length: 1092608 target oid 41f9c9be05ebe07af296a8b4ee79140ead8d8872e9243d104271c986ad427ed2 offset: 0 snap 30 2024-04-20T02:52:34.396 INFO:tasks.rados.rados.0.smithi105.stdout:1612: write (excl) oid 28 current snap is 33 2024-04-20T02:52:34.396 INFO:tasks.rados.rados.0.smithi105.stdout:1597: finishing set_chunk to oid 16 2024-04-20T02:52:34.396 INFO:tasks.rados.rados.0.smithi105.stdout:1597: got expected ENOENT (src dne) 2024-04-20T02:52:34.396 INFO:tasks.rados.rados.0.smithi105.stdout:1599: finishing rollback tid 0 to smithi10548823-40 2024-04-20T02:52:34.396 INFO:tasks.rados.rados.0.smithi105.stdout:1612: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2024-04-20T02:52:34.401 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:34.401 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:34.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:34.401 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:34.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.401 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:34.401 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:34.401 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 103.456864623, 2024-04-20T02:52:34.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:52:34.401 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:52:34.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:52:34.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:52:34.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 103.449386305, 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:52:34.407 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 103.44462840600001, 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:52:34.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 100.02119947200001, 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.418 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:52:34.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:52:34.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:52:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:52:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:52:34.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:52:34.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:52:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:52:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:52:34.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:52:34.427 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 100.021162589, 2024-04-20T02:52:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:52:34.428 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:34.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:34.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:34.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:52:34.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:52:34.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:52:34.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:52:34.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:52:34.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:52:34.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:52:34.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:52:34.440 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.440 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.440 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.441 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:34.441 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:34.441 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 100.021137956, 2024-04-20T02:52:34.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:52:34.441 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.441 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:34.441 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.441 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.441 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:34.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:52:34.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 100.014644545, 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.446 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:34.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:52:34.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:52:34.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:52:34.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:52:34.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:52:34.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:52:34.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:34.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:52:34.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:52:34.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.453 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 100.00858052300001, 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.454 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:52:34.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:52:34.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:52:34.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:52:34.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:52:34.460 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.460 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.460 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.461 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:34.461 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:34.461 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 99.998518758000003, 2024-04-20T02:52:34.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:52:34.461 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.461 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.462 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.462 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.462 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.462 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:52:34.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.462 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:52:34.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:52:34.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:52:34.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:52:34.468 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.468 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.468 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.469 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:34.469 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:34.469 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 99.997826153999995, 2024-04-20T02:52:34.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:52:34.469 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:34.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:52:34.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:52:34.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 99.997814500000004, 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.476 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:52:34.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.291749944000003, 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.481 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:34.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:52:34.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.272689228000004, 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:52:34.486 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:52:34.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.492 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:34.493 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:34.493 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.266742914000005, 2024-04-20T02:52:34.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:52:34.493 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.493 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.493 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.493 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.493 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.493 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:52:34.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:52:34.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:34.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:52:34.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:52:34.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:52:34.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:52:34.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:52:34.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:52:34.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.266177472999999, 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.500 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.506 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.266151441999995, 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:52:34.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:52:34.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:52:34.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:52:34.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:52:34.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 38.263938641000003, 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.513 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:34.514 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.514 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.514 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.514 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:52:34.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.514 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 38.258589117, 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.519 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:34.523 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 38.25811916, 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:34.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.528 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 38.258096623999997, 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:34.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:34.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:52:34.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:34.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:34.533 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:34.536 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:1612: writing smithi10548823-28 from 776556 to 1407368 tid 1 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:1612: writing smithi10548823-28 from 1823748 to 2476185 tid 2 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:1612: writing smithi10548823-28 from 3108517 to 3176556 tid 3 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:1597: done (15 left) 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:1598: done (14 left) 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:1613: read oid 36 snap -1 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:1613: expect (ObjNum 304 snap 33 seq_num 304) 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:1614: write (excl) oid 37 current snap is 33 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:1599: finishing rollback tid 1 to smithi10548823-40 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:1599: finishing rollback tid 2 to smithi10548823-40 2024-04-20T02:52:34.537 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 356 (ObjNum 248 snap 22 seq_num 248) dirty exists 2024-04-20T02:52:34.538 INFO:tasks.rados.rados.0.smithi105.stdout:1600: got (0) Success 2024-04-20T02:52:34.538 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 226 snap 22 seq_num 226) dirty exists 2024-04-20T02:52:34.538 INFO:tasks.rados.rados.0.smithi105.stdout:1602: finishing set_chunk to oid 9 2024-04-20T02:52:34.538 INFO:tasks.rados.rados.0.smithi105.stdout:1602: got expected ENOENT (src dne) 2024-04-20T02:52:34.538 INFO:tasks.rados.rados.0.smithi105.stdout:1601: expect (ObjNum 247 snap 22 seq_num 247) 2024-04-20T02:52:34.538 INFO:tasks.rados.rados.0.smithi105.stdout:1605: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:34.538 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:34.535+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:34.636 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:34.635+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:34.708 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:34.708 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:34.708 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:34.708 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:34.718 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:34.757 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:34.757+0000 7f03997757c0 -1 osd.2 121 log_to_monitors true 2024-04-20T02:52:34.759 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:34.759+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:34.767 INFO:tasks.rados.rados.0.smithi105.stdout:1603: expect (ObjNum 223 snap 22 seq_num 223) 2024-04-20T02:52:34.838 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:34.837+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:34.884 INFO:tasks.rados.rados.0.smithi105.stdout:1604: expect (ObjNum 289 snap 30 seq_num 289) 2024-04-20T02:52:34.886 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:34.886 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:34.886 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:34.887 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:34.887 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:34.897 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:34.938 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:34.938+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:35.039 INFO:tasks.rados.rados.0.smithi105.stdout:1608: got (22) Invalid argument 2024-04-20T02:52:35.040 INFO:tasks.rados.rados.0.smithi105.stdout:1607: expect (ObjNum 232 snap 22 seq_num 232) 2024-04-20T02:52:35.040 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:35.039+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:35.068 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:35.068 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.331778504, 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.069 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:52:35.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:35.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:52:35.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:52:35.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:52:35.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:52:35.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:52:35.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:52:35.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:52:35.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:52:35.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:52:35.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:52:35.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:52:35.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:52:35.077 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.077 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.077 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.077 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:35.078 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:35.078 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.300489443, 2024-04-20T02:52:35.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:52:35.078 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.078 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.078 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.078 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.078 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.078 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:35.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:52:35.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:52:35.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:52:35.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:52:35.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:52:35.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.295798987, 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.084 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:52:35.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.289042518, 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:52:35.090 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:35.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:52:35.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:35.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.097 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:35.097 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:35.097 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.279227007, 2024-04-20T02:52:35.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:52:35.097 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:52:35.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:52:35.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.103 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.278183075, 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.104 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:52:35.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:52:35.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:52:35.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.109 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.278161915, 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:35.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:52:35.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:52:35.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:52:35.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:35.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:52:35.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:52:35.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:52:35.114 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.070591334, 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:52:35.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:52:35.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:52:35.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:52:35.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:35.119 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.056396689, 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:52:35.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:52:35.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:52:35.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:52:35.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:52:35.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.050429432, 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:52:35.125 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:52:35.129 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.042023429, 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:52:35.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:52:35.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.938391388, 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:52:35.138 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.707795078, 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.680850929, 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.143 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:52:35.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:52:35.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:35.147 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.655492662, 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:52:35.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.151 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.627352218, 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:52:35.155 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.589983613, 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:35.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.58968668999999, 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.160 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:35.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:35.164 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.58460357200001, 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:52:35.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:52:35.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.168 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.557601013, 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:52:35.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:52:35.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:52:35.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:52:35.173 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.173 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.173 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.173 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.173 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.173 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:35.174 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:35.175 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:35.174+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:35.247 INFO:tasks.rados.rados.0.smithi105.stdout:1609: finishing set_chunk to oid 43 2024-04-20T02:52:35.247 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 43 set_chunk returned error code -95 offset: 145408 length: 2260480 2024-04-20T02:52:35.247 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 0 (ObjNum 276 snap 26 seq_num 276) dirty exists 2024-04-20T02:52:35.248 INFO:tasks.rados.rados.0.smithi105.stdout:1611: finishing set_chunk to oid 5 2024-04-20T02:52:35.248 INFO:tasks.rados.rados.0.smithi105.stdout:1613: expect (ObjNum 304 snap 33 seq_num 304) 2024-04-20T02:52:35.275 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:35.274+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:35.289 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:35.328 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:35.328+0000 7f038970f700 -1 osd.2 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:52:35.345 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:35.346 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:35.346 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:35.346 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:35.356 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:35.376 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:35.376+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:35.404 INFO:tasks.rados.rados.0.smithi105.stdout:1612: finishing write tid 1 to smithi10548823-28 2024-04-20T02:52:35.404 INFO:tasks.rados.rados.0.smithi105.stdout:1612: finishing write tid 2 to smithi10548823-28 2024-04-20T02:52:35.404 INFO:tasks.rados.rados.0.smithi105.stdout:1612: finishing write tid 3 to smithi10548823-28 2024-04-20T02:52:35.404 INFO:tasks.rados.rados.0.smithi105.stdout:1614: seq_num 309 ranges {793363=706061,2282230=796095} 2024-04-20T02:52:35.412 INFO:tasks.rados.rados.0.smithi105.stdout:1614: writing smithi10548823-37 from 793363 to 1499424 tid 1 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1614: writing smithi10548823-37 from 2282230 to 3078325 tid 2 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1599: done (15 left) 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1600: done (14 left) 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1601: done (13 left) 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1602: done (12 left) 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1603: done (11 left) 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1604: done (10 left) 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1605: done (9 left) 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1606: done (8 left) 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1607: done (7 left) 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1608: done (6 left) 2024-04-20T02:52:35.423 INFO:tasks.rados.rados.0.smithi105.stdout:1609: done (5 left) 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1610: done (4 left) 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1611: done (3 left) 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1613: done (2 left) 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1615: read oid 17 snap 28 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1615: expect deleted 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1616: read oid 8 snap -1 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1616: expect (ObjNum 290 snap 31 seq_num 290) 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1617: read oid 3 snap -1 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1617: expect deleted 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1618: delete oid 6 current snap is 33 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1612: finishing write tid 4 to smithi10548823-28 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1612: finishing write tid 5 to smithi10548823-28 2024-04-20T02:52:35.424 INFO:tasks.rados.rados.0.smithi105.stdout:1612: oid 28 updating version 0 to 191 2024-04-20T02:52:35.425 INFO:tasks.rados.rados.0.smithi105.stdout:1612: oid 28 updating version 191 to 194 2024-04-20T02:52:35.425 INFO:tasks.rados.rados.0.smithi105.stdout:1612: oid 28 version 194 is already newer than 192 2024-04-20T02:52:35.425 INFO:tasks.rados.rados.0.smithi105.stdout:1612: oid 28 version 194 is already newer than 193 2024-04-20T02:52:35.425 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 194 (ObjNum 308 snap 33 seq_num 308) dirty exists 2024-04-20T02:52:35.425 INFO:tasks.rados.rados.0.smithi105.stdout:1612: left oid 28 (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:52:35.425 INFO:tasks.rados.rados.0.smithi105.stdout:1612: done (5 left) 2024-04-20T02:52:35.425 INFO:tasks.rados.rados.0.smithi105.stdout:1615: done (4 left) 2024-04-20T02:52:35.425 INFO:tasks.rados.rados.0.smithi105.stdout:1617: done (3 left) 2024-04-20T02:52:35.425 INFO:tasks.rados.rados.0.smithi105.stdout:1618: done (2 left) 2024-04-20T02:52:35.425 INFO:tasks.rados.rados.0.smithi105.stdout:1619: set_chunk oid 30 target oid 2024-04-20T02:52:35.462 INFO:tasks.rados.rados.0.smithi105.stdout:1619: set_chunk oid 30 offset: 1700864 length: 1088512 target oid a17e9a45584abbd90252d528373d6b72eab66c898d68e1b96d4710d630728894 offset: 0 snap -1 2024-04-20T02:52:35.462 INFO:tasks.rados.rados.0.smithi105.stdout:1620: tier_evict oid 46 2024-04-20T02:52:35.462 INFO:tasks.rados.rados.0.smithi105.stdout:1620: tier_evict oid 46 snap -1 2024-04-20T02:52:35.462 INFO:tasks.rados.rados.0.smithi105.stdout:1616: expect (ObjNum 290 snap 31 seq_num 290) 2024-04-20T02:52:35.468 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:35.477 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:35.476+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:35.488 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:35.507 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:35.532 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:35.532 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:35.533 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:35.533 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:35.533 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:35.543 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:35.584 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:35.584+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:35.584 INFO:tasks.rados.rados.0.smithi105.stdout:1619: finishing set_chunk to oid 30 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 1700864 length: 1088512 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 226 snap 22 seq_num 226) dirty exists 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:1620: got (22) Invalid argument 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:1614: finishing write tid 1 to smithi10548823-37 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:1614: finishing write tid 2 to smithi10548823-37 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:1614: finishing write tid 3 to smithi10548823-37 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:1614: finishing write tid 4 to smithi10548823-37 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:1614: oid 37 updating version 0 to 77 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:1614: oid 37 version 77 is already newer than 75 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:1614: oid 37 version 77 is already newer than 76 2024-04-20T02:52:35.585 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 77 (ObjNum 309 snap 33 seq_num 309) dirty exists 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1614: left oid 37 (ObjNum 309 snap 33 seq_num 309) 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1614: done (3 left) 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1616: done (2 left) 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1619: done (1 left) 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1620: done (0 left) 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1621: rollback oid 15 current snap is 33 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 15 to 32 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1622: read oid 26 snap 32 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1622: expect (ObjNum 223 snap 22 seq_num 223) 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1623: tier_evict oid 29 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1623: tier_evict oid 29 snap -1 2024-04-20T02:52:35.586 INFO:tasks.rados.rados.0.smithi105.stdout:1624: set_chunk oid 14 target oid 2024-04-20T02:52:35.617 INFO:tasks.rados.rados.0.smithi105.stdout:1624: set_chunk oid 14 offset: 1051648 length: 1359872 target oid c7299e646ed40ee88954421237f7a1c5281d91008f88a85c8ea9f08d11ce54c5 offset: 0 snap -1 2024-04-20T02:52:35.617 INFO:tasks.rados.rados.0.smithi105.stdout:1625: write oid 19 current snap is 33 2024-04-20T02:52:35.617 INFO:tasks.rados.rados.0.smithi105.stdout:1623: got (22) Invalid argument 2024-04-20T02:52:35.618 INFO:tasks.rados.rados.0.smithi105.stdout:1625: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-04-20T02:52:35.621 INFO:tasks.rados.rados.0.smithi105.stdout:1625: writing smithi10548823-19 from 426977 to 883536 tid 1 2024-04-20T02:52:35.629 INFO:tasks.rados.rados.0.smithi105.stdout:1625: writing smithi10548823-19 from 1482841 to 2282605 tid 2 2024-04-20T02:52:35.634 INFO:tasks.rados.rados.0.smithi105.stdout:1625: writing smithi10548823-19 from 2958080 to 3226977 tid 3 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1623: done (4 left) 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1626: read oid 5 snap 28 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1626: expect (ObjNum 262 snap 23 seq_num 262) 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1627: read oid 20 snap -1 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1627: expect (ObjNum 289 snap 30 seq_num 289) 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1628: read oid 35 snap -1 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1628: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1629: read oid 7 snap -1 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1629: expect (ObjNum 295 snap 32 seq_num 295) 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1630: read oid 36 snap 30 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1630: expect (ObjNum 244 snap 22 seq_num 244) 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1631: tier_promote oid 37 2024-04-20T02:52:35.635 INFO:tasks.rados.rados.0.smithi105.stdout:1632: set_chunk oid 48 target oid 2024-04-20T02:52:35.655 INFO:tasks.rados.rados.0.smithi105.stdout:1632: set_chunk oid 48 offset: 572928 length: 453632 target oid f880eeb04a8c75c837d53eeecae71e3e8e2fb3d2277d03c12ab12e97b41e80ec offset: 0 snap -1 2024-04-20T02:52:35.656 INFO:tasks.rados.rados.0.smithi105.stdout:1633: read oid 44 snap 28 2024-04-20T02:52:35.656 INFO:tasks.rados.rados.0.smithi105.stdout:1633: expect (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:52:35.656 INFO:tasks.rados.rados.0.smithi105.stdout:1634: rollback oid 12 current snap is 33 2024-04-20T02:52:35.656 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 12 to 32 2024-04-20T02:52:35.656 INFO:tasks.rados.rados.0.smithi105.stdout:1621: finishing rollback tid 1 to smithi10548823-15 2024-04-20T02:52:35.656 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 306 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-20T02:52:35.656 INFO:tasks.rados.rados.0.smithi105.stdout:1622: expect (ObjNum 223 snap 22 seq_num 223) 2024-04-20T02:52:35.679 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:35.680+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 136.12737179999999, 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.732 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:52:35.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:52:35.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:52:35.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:52:35.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.738 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 114.363505401, 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:35.742 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 114.25020236, 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:52:35.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:52:35.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:35.748 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 114.244436843, 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:52:35.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:52:35.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:52:35.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:52:35.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:52:35.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:52:35.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:35.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:52:35.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:52:35.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.890416708, 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:52:35.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:52:35.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:52:35.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.88005310299999, 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:52:35.759 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.871968624, 2024-04-20T02:52:35.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:52:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:52:35.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:52:35.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 95.590410399000007, 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.768 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:52:35.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:52:35.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:52:35.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:52:35.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:52:35.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:52:35.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:35.774 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 95.585273289, 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:52:35.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:52:35.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:52:35.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.780 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 95.585000277000006, 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.781 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:52:35.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:52:35.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:52:35.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:52:35.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:52:35.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:52:35.787 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.787 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.787 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 95.584970892000001, 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:35.792 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 85.276076927000005, 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035060916999999997, 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456603+0000", 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001394295 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456616+0000", 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2605999999999999e-05 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456631+0000", 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4735e-05 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456682+0000", 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1048000000000003e-05 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456705+0000", 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2881e-05 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490235+0000", 2024-04-20T02:52:35.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033530250999999997 2024-04-20T02:52:35.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490270+0000", 2024-04-20T02:52:35.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5379000000000001e-05 2024-04-20T02:52:35.796 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.796 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 72.897643592999998, 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2132 4.10 e70/63 4:0b0a7da5:::smithi10548823-43:head v 70'76, mlcod=65'73)", 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 67.987045554000005, 2024-04-20T02:52:35.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036823724000000002, 2024-04-20T02:52:35.802 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.802 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.802 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:35.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744239+0000", 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744631+0000", 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039175199999999999 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744644+0000", 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3079e-05 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744653+0000", 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8629999999999997e-06 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744706+0000", 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2627999999999998e-05 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744737+0000", 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1387999999999998e-05 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781039+0000", 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036302147 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781064+0000", 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5182999999999999e-05 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2293 4.3 e76/60 4:c07c6daa:::smithi10548823-7:head v 76'165, mlcod=64'161)", 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.542177531, 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034106808000000002, 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189108+0000", 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190758+0000", 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016504740000000001 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190778+0000", 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9507000000000001e-05 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190787+0000", 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1740000000000006e-06 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190836+0000", 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9351999999999998e-05 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190854+0000", 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7629e-05 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.223165+0000", 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032310885999999997 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.223216+0000", 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0615000000000002e-05 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.809 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 47.450163371000002, 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.813 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 47.432894937, 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:52:35.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:52:35.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:52:35.817 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.818 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.818 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.818 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.14 98/97 [PushOp(4:2e88ea27:::smithi10548823-12:head, version: 98'33, data_included: [406769~907695,1439434~140881], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:2e88ea27:::smithi10548823-12:head@98'33, size: 2033680, copy_subset: [0~2033680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1580315, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:52:35.818 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:35.818 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 39.481128103000003, 2024-04-20T02:52:35.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033699081999999998, 2024-04-20T02:52:35.818 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.818 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:35.819 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:35.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251322+0000", 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001164251 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251340+0000", 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.789e-05 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251356+0000", 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5962000000000001e-05 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251428+0000", 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2750000000000007e-05 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251444+0000", 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5248e-05 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.283857+0000", 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032413659999999997 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.6.0:82 4.15 e104/103 4:a9897a5e:::smithi10548823-42:head v 104'64, mlcod=93'62)", 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:35.822 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.595413809, 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035573920000000002, 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135871+0000", 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136395+0000", 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00052370100000000001 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136409+0000", 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4191e-05 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136418+0000", 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0100000000000001e-06 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136468+0000", 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9458e-05 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136499+0000", 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1062999999999997e-05 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.171417+0000", 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034918607999999997 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.171446+0000", 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9000999999999999e-05 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:35.826 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 24.259749142, 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:35.830 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:35.832 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:35.832 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:35.832 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:35.833 INFO:tasks.rados.rados.0.smithi105.stdout:1624: finishing set_chunk to oid 14 2024-04-20T02:52:35.833 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk returned error code -95 offset: 1051648 length: 1359872 2024-04-20T02:52:35.833 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 274 snap 26 seq_num 274) dirty exists 2024-04-20T02:52:35.833 INFO:tasks.rados.rados.0.smithi105.stdout:1627: expect (ObjNum 289 snap 30 seq_num 289) 2024-04-20T02:52:35.833 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:35.834 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:35.834 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:35.832+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:35.835 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:35.835 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:35.835 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-20T02:52:35.835 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:35.835 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:35.835 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:35.835 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:35.835 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:35.836 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:35.932 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:35.932+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:35.944 INFO:tasks.rados.rados.0.smithi105.stdout:1628: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:52:36.003 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:36.004 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:36.010 INFO:tasks.rados.rados.0.smithi105.stdout:1629: expect (ObjNum 295 snap 32 seq_num 295) 2024-04-20T02:52:36.014 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:36.058 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:36.058+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:36.065 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:36.134 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:36.133+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:36.143 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:36.144 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:36.154 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:36.159 INFO:tasks.rados.rados.0.smithi105.stdout:1631: got (0) Success 2024-04-20T02:52:36.160 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 77 (ObjNum 309 snap 33 seq_num 309) dirty exists 2024-04-20T02:52:36.160 INFO:tasks.rados.rados.0.smithi105.stdout:1621: done (12 left) 2024-04-20T02:52:36.160 INFO:tasks.rados.rados.0.smithi105.stdout:1622: done (11 left) 2024-04-20T02:52:36.160 INFO:tasks.rados.rados.0.smithi105.stdout:1624: done (10 left) 2024-04-20T02:52:36.160 INFO:tasks.rados.rados.0.smithi105.stdout:1627: done (9 left) 2024-04-20T02:52:36.160 INFO:tasks.rados.rados.0.smithi105.stdout:1628: done (8 left) 2024-04-20T02:52:36.160 INFO:tasks.rados.rados.0.smithi105.stdout:1629: done (7 left) 2024-04-20T02:52:36.160 INFO:tasks.rados.rados.0.smithi105.stdout:1631: done (6 left) 2024-04-20T02:52:36.160 INFO:tasks.rados.rados.0.smithi105.stdout:1635: write (excl) oid 24 current snap is 33 2024-04-20T02:52:36.160 INFO:tasks.rados.rados.0.smithi105.stdout:1634: finishing rollback tid 1 to smithi10548823-12 2024-04-20T02:52:36.161 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 37 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-20T02:52:36.161 INFO:tasks.rados.rados.0.smithi105.stdout:1632: finishing set_chunk to oid 48 2024-04-20T02:52:36.161 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 67 (ObjNum 195 snap 19 seq_num 195) dirty exists 2024-04-20T02:52:36.161 INFO:tasks.rados.rados.0.smithi105.stdout:1633: expect (ObjNum 208 snap 19 seq_num 208) 2024-04-20T02:52:36.201 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:36.201 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:36.201 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:36.201 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:36.211 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-04-20T02:52:36.239 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:36.239+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:36.326 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:36.326 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:36.327 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:52:36.336 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:36.371 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:36.371+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:36.383 INFO:tasks.rados.rados.0.smithi105.stdout:1626: expect (ObjNum 262 snap 23 seq_num 262) 2024-04-20T02:52:36.437 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:36.436+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:36.505 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:36.506 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:36.506 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:36.506 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:36.516 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:36.519 INFO:tasks.rados.rados.0.smithi105.stdout:1630: expect (ObjNum 244 snap 22 seq_num 244) 2024-04-20T02:52:36.562 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:36.562+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:1625: finishing write tid 1 to smithi10548823-19 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:1625: finishing write tid 2 to smithi10548823-19 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:1625: finishing write tid 3 to smithi10548823-19 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:1625: finishing write tid 4 to smithi10548823-19 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:1625: finishing write tid 5 to smithi10548823-19 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:1625: oid 19 updating version 0 to 126 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:1625: oid 19 version 126 is already newer than 125 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:1625: oid 19 updating version 126 to 127 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:1625: oid 19 version 127 is already newer than 124 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 127 (ObjNum 311 snap 33 seq_num 311) dirty exists 2024-04-20T02:52:36.603 INFO:tasks.rados.rados.0.smithi105.stdout:1625: left oid 19 (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:52:36.604 INFO:tasks.rados.rados.0.smithi105.stdout:1635: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-04-20T02:52:36.608 INFO:tasks.rados.rados.0.smithi105.stdout:1635: writing smithi10548823-24 from 443784 to 1091945 tid 1 2024-04-20T02:52:36.615 INFO:tasks.rados.rados.0.smithi105.stdout:1635: writing smithi10548823-24 from 1657676 to 2317451 tid 2 2024-04-20T02:52:36.621 INFO:tasks.rados.rados.0.smithi105.stdout:1635: writing smithi10548823-24 from 2818209 to 3243784 tid 3 2024-04-20T02:52:36.621 INFO:tasks.rados.rados.0.smithi105.stdout:1625: done (6 left) 2024-04-20T02:52:36.621 INFO:tasks.rados.rados.0.smithi105.stdout:1626: done (5 left) 2024-04-20T02:52:36.622 INFO:tasks.rados.rados.0.smithi105.stdout:1630: done (4 left) 2024-04-20T02:52:36.622 INFO:tasks.rados.rados.0.smithi105.stdout:1632: done (3 left) 2024-04-20T02:52:36.622 INFO:tasks.rados.rados.0.smithi105.stdout:1633: done (2 left) 2024-04-20T02:52:36.622 INFO:tasks.rados.rados.0.smithi105.stdout:1634: done (1 left) 2024-04-20T02:52:36.622 INFO:tasks.rados.rados.0.smithi105.stdout:1636: set_chunk oid 9 target oid 2024-04-20T02:52:36.622 INFO:tasks.rados.rados.0.smithi105.stdout:1636: set_chunk oid 9 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:36.622 INFO:tasks.rados.rados.0.smithi105.stdout:1637: read oid 28 snap 28 2024-04-20T02:52:36.622 INFO:tasks.rados.rados.0.smithi105.stdout:1637: expect (ObjNum 265 snap 25 seq_num 265) 2024-04-20T02:52:36.622 INFO:tasks.rados.rados.0.smithi105.stdout:1638: write (excl) oid 21 current snap is 33 2024-04-20T02:52:36.622 INFO:tasks.rados.rados.0.smithi105.stdout:1638: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-04-20T02:52:36.628 INFO:tasks.rados.rados.0.smithi105.stdout:1638: writing smithi10548823-21 from 460591 to 1184001 tid 1 2024-04-20T02:52:36.633 INFO:tasks.rados.rados.0.smithi105.stdout:1638: writing smithi10548823-21 from 1716158 to 2119591 tid 2 2024-04-20T02:52:36.639 INFO:tasks.rados.rados.0.smithi105.stdout:1638: writing smithi10548823-21 from 2729279 to 3169419 tid 3 2024-04-20T02:52:36.640 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:36.640+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:36.650 INFO:tasks.rados.rados.0.smithi105.stdout:1639: write (excl) oid 44 current snap is 33 2024-04-20T02:52:36.651 INFO:tasks.rados.rados.0.smithi105.stdout:1639: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-04-20T02:52:36.657 INFO:tasks.rados.rados.0.smithi105.stdout:1639: writing smithi10548823-44 from 477398 to 1276057 tid 1 2024-04-20T02:52:36.665 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:36.665 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:36.665 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:36.665 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:36.665 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:36.665 INFO:tasks.rados.rados.0.smithi105.stdout:1639: writing smithi10548823-44 from 1774640 to 2438084 tid 2 2024-04-20T02:52:36.670 INFO:tasks.rados.rados.0.smithi105.stdout:1639: writing smithi10548823-44 from 2873055 to 3277398 tid 3 2024-04-20T02:52:36.670 INFO:tasks.rados.rados.0.smithi105.stdout:1640: write (excl) oid 49 current snap is 33 2024-04-20T02:52:36.670 INFO:tasks.rados.rados.0.smithi105.stdout:1640: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-04-20T02:52:36.677 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:36.679 INFO:tasks.rados.rados.0.smithi105.stdout:1640: writing smithi10548823-49 from 494205 to 968113 tid 1 2024-04-20T02:52:36.717 INFO:tasks.rados.rados.0.smithi105.stdout:1640: writing smithi10548823-49 from 1433122 to 1840224 tid 2 2024-04-20T02:52:36.717 INFO:tasks.rados.rados.0.smithi105.stdout:1640: writing smithi10548823-49 from 2384125 to 2846809 tid 3 2024-04-20T02:52:36.717 INFO:tasks.rados.rados.0.smithi105.stdout:1641: read oid 20 snap -1 2024-04-20T02:52:36.717 INFO:tasks.rados.rados.0.smithi105.stdout:1641: expect (ObjNum 289 snap 30 seq_num 289) 2024-04-20T02:52:36.717 INFO:tasks.rados.rados.0.smithi105.stdout:1642: write (excl) oid 46 current snap is 33 2024-04-20T02:52:36.717 INFO:tasks.rados.rados.0.smithi105.stdout:1636: finishing set_chunk to oid 9 2024-04-20T02:52:36.717 INFO:tasks.rados.rados.0.smithi105.stdout:1636: got expected ENOENT (src dne) 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1642: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1642: writing smithi10548823-46 from 511012 to 1060169 tid 1 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1642: writing smithi10548823-46 from 1775251 to 2326011 tid 2 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1642: writing smithi10548823-46 from 3095195 to 3311012 tid 3 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1636: done (7 left) 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1643: read oid 39 snap -1 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1643: expect (ObjNum 283 snap 27 seq_num 283) 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1644: read oid 31 snap -1 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1644: expect (ObjNum 305 snap 33 seq_num 305) 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1645: snap_create 2024-04-20T02:52:36.718 INFO:tasks.rados.rados.0.smithi105.stdout:1637: expect (ObjNum 265 snap 25 seq_num 265) 2024-04-20T02:52:36.740 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:36.739+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:36.840 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:36.840+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:36.861 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:36.862 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:36.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:36.862 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-04-20T02:52:36.862 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:36.872 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:36.913 INFO:tasks.rados.rados.0.smithi105.stdout:1635: finishing write tid 1 to smithi10548823-24 2024-04-20T02:52:36.913 INFO:tasks.rados.rados.0.smithi105.stdout:1635: finishing write tid 2 to smithi10548823-24 2024-04-20T02:52:36.913 INFO:tasks.rados.rados.0.smithi105.stdout:1635: finishing write tid 3 to smithi10548823-24 2024-04-20T02:52:36.914 INFO:tasks.rados.rados.0.smithi105.stdout:1635: finishing write tid 4 to smithi10548823-24 2024-04-20T02:52:36.914 INFO:tasks.rados.rados.0.smithi105.stdout:1635: finishing write tid 5 to smithi10548823-24 2024-04-20T02:52:36.914 INFO:tasks.rados.rados.0.smithi105.stdout:1635: oid 24 updating version 0 to 360 2024-04-20T02:52:36.914 INFO:tasks.rados.rados.0.smithi105.stdout:1635: oid 24 version 360 is already newer than 359 2024-04-20T02:52:36.914 INFO:tasks.rados.rados.0.smithi105.stdout:1635: oid 24 updating version 360 to 361 2024-04-20T02:52:36.914 INFO:tasks.rados.rados.0.smithi105.stdout:1635: oid 24 version 361 is already newer than 358 2024-04-20T02:52:36.914 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 361 (ObjNum 312 snap 33 seq_num 312) dirty exists 2024-04-20T02:52:36.914 INFO:tasks.rados.rados.0.smithi105.stdout:1635: left oid 24 (ObjNum 312 snap 33 seq_num 312) 2024-04-20T02:52:36.914 INFO:tasks.rados.rados.0.smithi105.stdout:1641: expect (ObjNum 289 snap 30 seq_num 289) 2024-04-20T02:52:36.941 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:36.941+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:37.042 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:37.042+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:37.043 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:37.044 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:37.054 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:37.077 INFO:tasks.rados.rados.0.smithi105.stdout:1638: finishing write tid 1 to smithi10548823-21 2024-04-20T02:52:37.077 INFO:tasks.rados.rados.0.smithi105.stdout:1638: finishing write tid 2 to smithi10548823-21 2024-04-20T02:52:37.077 INFO:tasks.rados.rados.0.smithi105.stdout:1638: finishing write tid 3 to smithi10548823-21 2024-04-20T02:52:37.077 INFO:tasks.rados.rados.0.smithi105.stdout:1638: finishing write tid 4 to smithi10548823-21 2024-04-20T02:52:37.077 INFO:tasks.rados.rados.0.smithi105.stdout:1638: finishing write tid 5 to smithi10548823-21 2024-04-20T02:52:37.077 INFO:tasks.rados.rados.0.smithi105.stdout:1638: oid 21 updating version 0 to 158 2024-04-20T02:52:37.078 INFO:tasks.rados.rados.0.smithi105.stdout:1638: oid 21 version 158 is already newer than 157 2024-04-20T02:52:37.078 INFO:tasks.rados.rados.0.smithi105.stdout:1638: oid 21 version 158 is already newer than 156 2024-04-20T02:52:37.078 INFO:tasks.rados.rados.0.smithi105.stdout:1638: oid 21 version 158 is already newer than 155 2024-04-20T02:52:37.078 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 158 (ObjNum 313 snap 33 seq_num 313) dirty exists 2024-04-20T02:52:37.078 INFO:tasks.rados.rados.0.smithi105.stdout:1638: left oid 21 (ObjNum 313 snap 33 seq_num 313) 2024-04-20T02:52:37.078 INFO:tasks.rados.rados.0.smithi105.stdout:1643: expect (ObjNum 283 snap 27 seq_num 283) 2024-04-20T02:52:37.143 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:37.143+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:37.144 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:52:37.214 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:37.214 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:37.225 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:37.244 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:37.244+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1640: finishing write tid 1 to smithi10548823-49 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1640: finishing write tid 2 to smithi10548823-49 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1635: done (9 left) 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1637: done (8 left) 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1638: done (7 left) 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1641: done (6 left) 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1643: done (5 left) 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1645: done (4 left) 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1646: write (excl) oid 27 current snap is 34 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1640: finishing write tid 3 to smithi10548823-49 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1640: finishing write tid 4 to smithi10548823-49 2024-04-20T02:52:37.258 INFO:tasks.rados.rados.0.smithi105.stdout:1640: finishing write tid 5 to smithi10548823-49 2024-04-20T02:52:37.259 INFO:tasks.rados.rados.0.smithi105.stdout:1640: oid 49 updating version 0 to 215 2024-04-20T02:52:37.259 INFO:tasks.rados.rados.0.smithi105.stdout:1640: oid 49 updating version 215 to 216 2024-04-20T02:52:37.259 INFO:tasks.rados.rados.0.smithi105.stdout:1640: oid 49 updating version 216 to 217 2024-04-20T02:52:37.259 INFO:tasks.rados.rados.0.smithi105.stdout:1640: oid 49 updating version 217 to 218 2024-04-20T02:52:37.259 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 218 (ObjNum 315 snap 33 seq_num 315) dirty exists 2024-04-20T02:52:37.259 INFO:tasks.rados.rados.0.smithi105.stdout:1640: left oid 49 (ObjNum 315 snap 33 seq_num 315) 2024-04-20T02:52:37.259 INFO:tasks.rados.rados.0.smithi105.stdout:1644: expect (ObjNum 305 snap 33 seq_num 305) 2024-04-20T02:52:37.345 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:37.345+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:37.355 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:37.355 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:37.366 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:37.450 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:37.450+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:37.496 INFO:tasks.rados.rados.0.smithi105.stdout:1642: finishing write tid 1 to smithi10548823-46 2024-04-20T02:52:37.496 INFO:tasks.rados.rados.0.smithi105.stdout:1642: finishing write tid 2 to smithi10548823-46 2024-04-20T02:52:37.496 INFO:tasks.rados.rados.0.smithi105.stdout:1642: finishing write tid 3 to smithi10548823-46 2024-04-20T02:52:37.496 INFO:tasks.rados.rados.0.smithi105.stdout:1642: finishing write tid 4 to smithi10548823-46 2024-04-20T02:52:37.497 INFO:tasks.rados.rados.0.smithi105.stdout:1642: finishing write tid 5 to smithi10548823-46 2024-04-20T02:52:37.497 INFO:tasks.rados.rados.0.smithi105.stdout:1642: oid 46 updating version 0 to 92 2024-04-20T02:52:37.497 INFO:tasks.rados.rados.0.smithi105.stdout:1642: oid 46 updating version 92 to 93 2024-04-20T02:52:37.497 INFO:tasks.rados.rados.0.smithi105.stdout:1642: oid 46 version 93 is already newer than 90 2024-04-20T02:52:37.497 INFO:tasks.rados.rados.0.smithi105.stdout:1642: oid 46 version 93 is already newer than 91 2024-04-20T02:52:37.497 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 93 (ObjNum 316 snap 33 seq_num 316) dirty exists 2024-04-20T02:52:37.497 INFO:tasks.rados.rados.0.smithi105.stdout:1642: left oid 46 (ObjNum 316 snap 33 seq_num 316) 2024-04-20T02:52:37.497 INFO:tasks.rados.rados.0.smithi105.stdout:1639: finishing write tid 1 to smithi10548823-44 2024-04-20T02:52:37.497 INFO:tasks.rados.rados.0.smithi105.stdout:1646: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-04-20T02:52:37.502 INFO:tasks.rados.rados.0.smithi105.stdout:1646: writing smithi10548823-27 from 527819 to 1152225 tid 1 2024-04-20T02:52:37.509 INFO:tasks.rados.rados.0.smithi105.stdout:1646: writing smithi10548823-27 from 1833733 to 2244504 tid 2 2024-04-20T02:52:37.517 INFO:tasks.rados.rados.0.smithi105.stdout:1646: writing smithi10548823-27 from 2722618 to 3324199 tid 3 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1640: done (4 left) 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1642: done (3 left) 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1644: done (2 left) 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1647: delete oid 48 current snap is 34 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1639: finishing write tid 2 to smithi10548823-44 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1639: finishing write tid 3 to smithi10548823-44 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1639: finishing write tid 4 to smithi10548823-44 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1639: finishing write tid 5 to smithi10548823-44 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1639: oid 44 updating version 0 to 112 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1639: oid 44 updating version 112 to 113 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1639: oid 44 version 113 is already newer than 111 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1639: oid 44 version 113 is already newer than 110 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 113 (ObjNum 314 snap 33 seq_num 314) dirty exists 2024-04-20T02:52:37.518 INFO:tasks.rados.rados.0.smithi105.stdout:1639: left oid 44 (ObjNum 314 snap 33 seq_num 314) 2024-04-20T02:52:37.520 INFO:tasks.rados.rados.0.smithi105.stdout:1639: done (2 left) 2024-04-20T02:52:37.520 INFO:tasks.rados.rados.0.smithi105.stdout:1647: done (1 left) 2024-04-20T02:52:37.520 INFO:tasks.rados.rados.0.smithi105.stdout:1648: set_chunk oid 18 target oid 2024-04-20T02:52:37.529 INFO:tasks.rados.rados.0.smithi105.stdout:1648: set_chunk oid 18 offset: 2627072 length: 318976 target oid d25615d61d3891a423258ec7866374b88f90efa4ca3bc955497ace6ec120eefe offset: 0 snap -1 2024-04-20T02:52:37.529 INFO:tasks.rados.rados.0.smithi105.stdout:1649: read oid 43 snap -1 2024-04-20T02:52:37.529 INFO:tasks.rados.rados.0.smithi105.stdout:1649: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:52:37.529 INFO:tasks.rados.rados.0.smithi105.stdout:1650: rollback oid 37 current snap is 34 2024-04-20T02:52:37.529 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 37 to 32 2024-04-20T02:52:37.529 INFO:tasks.rados.rados.0.smithi105.stdout:1651: snap_remove snap 28 2024-04-20T02:52:37.542 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:37.542 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:37.547 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:37.547+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:37.554 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:37.648 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:37.649+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:37.721 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:37.721 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:37.731 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:37.775 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:37.775+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:37.836 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:37.859 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:37.860 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:37.860 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:37.860+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:37.871 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:37.957 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:37.957+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:38.004 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:38.040 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:38.052 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:38.053+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:38.072 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:38.083 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:38.083 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:38.091 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:38.093 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:38.111 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:38.131 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:38.135 INFO:tasks.rados.rados.0.smithi105.stdout:1651: done (4 left) 2024-04-20T02:52:38.135 INFO:tasks.rados.rados.0.smithi105.stdout:1652: write oid 34 current snap is 34 2024-04-20T02:52:38.135 INFO:tasks.rados.rados.0.smithi105.stdout:1652: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-04-20T02:52:38.143 INFO:tasks.rados.rados.0.smithi105.stdout:1652: writing smithi10548823-34 from 561433 to 1336337 tid 1 2024-04-20T02:52:38.154 INFO:tasks.rados.rados.0.smithi105.stdout:1652: writing smithi10548823-34 from 1950697 to 2365137 tid 2 2024-04-20T02:52:38.154 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:38.153+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:38.157 INFO:tasks.rados.rados.0.smithi105.stdout:1652: writing smithi10548823-34 from 2777464 to 3361433 tid 3 2024-04-20T02:52:38.157 INFO:tasks.rados.rados.0.smithi105.stdout:1653: write (excl) oid 23 current snap is 34 2024-04-20T02:52:38.157 INFO:tasks.rados.rados.0.smithi105.stdout:1653: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-04-20T02:52:38.164 INFO:tasks.rados.rados.0.smithi105.stdout:1653: writing smithi10548823-23 from 578240 to 1144746 tid 1 2024-04-20T02:52:38.170 INFO:tasks.rados.rados.0.smithi105.stdout:1653: writing smithi10548823-23 from 1609179 to 2167277 tid 2 2024-04-20T02:52:38.178 INFO:tasks.rados.rados.0.smithi105.stdout:1653: writing smithi10548823-23 from 2804887 to 3356637 tid 3 2024-04-20T02:52:38.178 INFO:tasks.rados.rados.0.smithi105.stdout:1654: read oid 1 snap -1 2024-04-20T02:52:38.179 INFO:tasks.rados.rados.0.smithi105.stdout:1654: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:38.179 INFO:tasks.rados.rados.0.smithi105.stdout:1655: read oid 3 snap -1 2024-04-20T02:52:38.179 INFO:tasks.rados.rados.0.smithi105.stdout:1655: expect deleted 2024-04-20T02:52:38.179 INFO:tasks.rados.rados.0.smithi105.stdout:1656: read oid 29 snap -1 2024-04-20T02:52:38.179 INFO:tasks.rados.rados.0.smithi105.stdout:1656: expect (ObjNum 269 snap 25 seq_num 269) 2024-04-20T02:52:38.179 INFO:tasks.rados.rados.0.smithi105.stdout:1657: snap_create 2024-04-20T02:52:38.186 INFO:tasks.rados.rados.0.smithi105.stdout:1649: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:52:38.186 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:38.224 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:38.225 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:38.225 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:38.225 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:38.226 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:38.226 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:38.226 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:38.226 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:38.226 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:38.254 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:38.254+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:38.269 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:38.269 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:38.269 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:38.269 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:38.269 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:38.280 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:38.355 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:38.355+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:38.396 INFO:tasks.rados.rados.0.smithi105.stdout:1650: finishing rollback tid 0 to smithi10548823-37 2024-04-20T02:52:38.396 INFO:tasks.rados.rados.0.smithi105.stdout:1648: finishing set_chunk to oid 18 2024-04-20T02:52:38.396 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 100 (ObjNum 296 snap 32 seq_num 296) dirty exists 2024-04-20T02:52:38.396 INFO:tasks.rados.rados.0.smithi105.stdout:1650: finishing rollback tid 1 to smithi10548823-37 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1650: finishing rollback tid 2 to smithi10548823-37 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 83 (ObjNum 263 snap 24 seq_num 263) dirty exists 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1646: finishing write tid 1 to smithi10548823-27 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1646: finishing write tid 2 to smithi10548823-27 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1646: finishing write tid 3 to smithi10548823-27 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1646: finishing write tid 4 to smithi10548823-27 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1646: finishing write tid 5 to smithi10548823-27 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1646: oid 27 updating version 0 to 242 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1646: oid 27 version 242 is already newer than 241 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1646: oid 27 updating version 242 to 243 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1646: oid 27 version 243 is already newer than 240 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 243 (ObjNum 317 snap 34 seq_num 317) dirty exists 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1646: left oid 27 (ObjNum 317 snap 34 seq_num 317) 2024-04-20T02:52:38.397 INFO:tasks.rados.rados.0.smithi105.stdout:1656: expect (ObjNum 269 snap 25 seq_num 269) 2024-04-20T02:52:38.453 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:38.453 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:38.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:38.453 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 111.90878332299999, 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:38.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:52:38.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:52:38.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 111.902562249, 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:52:38.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:52:38.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:52:38.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 111.894162045, 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:38.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:52:38.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 111.8938194, 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:52:38.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:52:38.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.476 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 104.08749739700001, 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 104.08023443499999, 2024-04-20T02:52:38.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:52:38.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:52:38.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:52:38.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 104.074291906, 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.488 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.493 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 104.074088914, 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.494 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:52:38.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:52:38.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 104.074065987, 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:38.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:52:38.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:52:38.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:52:38.504 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 104.06398225300001, 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:52:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:52:38.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.508 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 104.047096845, 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:38.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:52:38.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:52:38.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.512 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 104.030820404, 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 89.006264931999993, 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:52:38.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:52:38.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.522 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.522 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:52:38.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:52:38.522 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.525 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.525 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 73.691949156000007, 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:52:38.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:52:38.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:52:38.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:52:38.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.531 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2545 4.9 e91/81 4:95e19a88:::smithi10548823-40:head v 91'253, mlcod=91'251)", 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.037312041, 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054757404000000003, 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:38.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415210+0000", 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416004+0000", 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079381199999999997 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416022+0000", 2024-04-20T02:52:38.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7969e-05 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416031+0000", 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9279999999999999e-06 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416100+0000", 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8891000000000002e-05 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416128+0000", 2024-04-20T02:52:38.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8203000000000001e-05 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469912+0000", 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053783794000000003 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469968+0000", 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6474000000000002e-05 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.104283234, 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:52:38.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2700 4.0 4:007b2a87:::smithi10548823-39:head [write 1967200~714747] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.095793039999997, 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061846287, 2024-04-20T02:52:38.541 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2700 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356728+0000", 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356879+0000", 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000150958 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356894+0000", 2024-04-20T02:52:38.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5132e-05 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356906+0000", 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2071e-05 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366666+0000", 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0097595289999999994 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366774+0000", 2024-04-20T02:52:38.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108527 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366956+0000", 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181359 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390515+0000", 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023559902000000001 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406790+0000", 2024-04-20T02:52:38.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016274555999999999 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418488+0000", 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011697591 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418529+0000", 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1545000000000003e-05 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418576+0000", 2024-04-20T02:52:38.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6903000000000001e-05 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2701 4.0 4:007b2a87:::smithi10548823-39:head [write 3421983~426820] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.088617048000003, 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054799176999999998, 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2701 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363904+0000", 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364154+0000", 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000250179 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364171+0000", 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6748e-05 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364182+0000", 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1371e-05 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390552+0000", 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026369607 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390652+0000", 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010059499999999999 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390857+0000", 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020445199999999999 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418224+0000", 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027367316999999999 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418282+0000", 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8279000000000002e-05 2024-04-20T02:52:38.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418618+0000", 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000335967 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418661+0000", 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2614999999999998e-05 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418705+0000", 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3993999999999999e-05 2024-04-20T02:52:38.552 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2702 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (57) in=7b,truncate 3848803] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.088330994000003, 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058669165000000002, 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2702 2024-04-20T02:52:38.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364191+0000", 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364196+0000", 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4490000000000001e-06 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364201+0000", 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5040000000000002e-06 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364206+0000", 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.405895+0000", 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041689047 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406021+0000", 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000125561 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:38.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406541+0000", 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052000699999999996 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418451+0000", 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011910041999999999 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418747+0000", 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029622399999999999 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:38.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422704+0000", 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039571859999999997 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422781+0000", 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.661e-05 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422861+0000", 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9987999999999994e-05 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.558 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2703 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.088311941999997, 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058940278999999998, 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2703 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364213+0000", 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4830000000000002e-06 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364220+0000", 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.007e-06 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364225+0000", 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0350000000000001e-06 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406820+0000", 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042595135999999999 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406923+0000", 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102846 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422897+0000", 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015974202999999999 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423015+0000", 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117581 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423151+0000", 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013635899999999999 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:38.563 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:38.565 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:38.565 INFO:tasks.rados.rados.0.smithi105.stdout:1652: finishing write tid 1 to smithi10548823-34 2024-04-20T02:52:38.565 INFO:tasks.rados.rados.0.smithi105.stdout:1654: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:38.566 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:38.565+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:38.629 INFO:tasks.rados.rados.0.smithi105.stdout:1652: finishing write tid 2 to smithi10548823-34 2024-04-20T02:52:38.629 INFO:tasks.rados.rados.0.smithi105.stdout:1652: finishing write tid 3 to smithi10548823-34 2024-04-20T02:52:38.629 INFO:tasks.rados.rados.0.smithi105.stdout:1652: finishing write tid 4 to smithi10548823-34 2024-04-20T02:52:38.629 INFO:tasks.rados.rados.0.smithi105.stdout:1652: finishing write tid 5 to smithi10548823-34 2024-04-20T02:52:38.629 INFO:tasks.rados.rados.0.smithi105.stdout:1652: oid 34 updating version 0 to 247 2024-04-20T02:52:38.629 INFO:tasks.rados.rados.0.smithi105.stdout:1652: oid 34 updating version 247 to 248 2024-04-20T02:52:38.629 INFO:tasks.rados.rados.0.smithi105.stdout:1652: oid 34 version 248 is already newer than 246 2024-04-20T02:52:38.629 INFO:tasks.rados.rados.0.smithi105.stdout:1652: oid 34 updating version 248 to 249 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 249 (ObjNum 319 snap 34 seq_num 319) dirty exists 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1652: left oid 34 (ObjNum 319 snap 34 seq_num 319) 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1653: finishing write tid 1 to smithi10548823-23 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1653: finishing write tid 2 to smithi10548823-23 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1653: finishing write tid 3 to smithi10548823-23 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1653: finishing write tid 4 to smithi10548823-23 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1653: finishing write tid 5 to smithi10548823-23 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1653: oid 23 updating version 0 to 120 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1653: oid 23 version 120 is already newer than 118 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1653: oid 23 version 120 is already newer than 119 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1653: oid 23 version 120 is already newer than 117 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 120 (ObjNum 320 snap 34 seq_num 320) dirty exists 2024-04-20T02:52:38.630 INFO:tasks.rados.rados.0.smithi105.stdout:1653: left oid 23 (ObjNum 320 snap 34 seq_num 320) 2024-04-20T02:52:38.666 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:38.667+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:38.707 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:38.707 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:38.707 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:38.707 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:38.717 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:38.766 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:38.767+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:38.845 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:38.845 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:38.845 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:38.845 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:38.845 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:38.855 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:38.896 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:38.896+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:38.968 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:38.969+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:39.024 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:39.024 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:39.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:39.024 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:39.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 108.080226201, 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:39.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:52:39.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:52:39.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:52:39.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.030 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 108.07274788300001, 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:52:39.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:52:39.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 108.06798998399999, 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.037 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.038 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:52:39.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.038 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:39.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:39.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.64456104999999, 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.042 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:52:39.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:52:39.048 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.049 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.049 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.049 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:39.049 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:39.049 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.644524167, 2024-04-20T02:52:39.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:52:39.049 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.050 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.050 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.050 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.050 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.050 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:52:39.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:39.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:52:39.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:52:39.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:52:39.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:52:39.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:52:39.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:39.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:52:39.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:52:39.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:52:39.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:52:39.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:52:39.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:52:39.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:52:39.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:52:39.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:52:39.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:39.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.644499534, 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.058 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:52:39.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:52:39.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:52:39.062 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.638006123, 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.063 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:39.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:52:39.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:52:39.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:52:39.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:52:39.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:52:39.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:52:39.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:52:39.069 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.63194210100001, 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.070 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:52:39.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:52:39.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:52:39.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.076 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:39.076 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:39.076 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.621880336, 2024-04-20T02:52:39.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:52:39.076 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.076 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.076 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.076 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.076 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.076 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.081 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.621187732, 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.082 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:39.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:52:39.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:52:39.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:52:39.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.621176078, 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.915111522000004, 2024-04-20T02:52:39.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:39.096 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.896050806000005, 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:52:39.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.890104492000006, 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.102 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.107 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.889539051, 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.889513019999995, 2024-04-20T02:52:39.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:39.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 42.887300218999997, 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.118 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:39.122 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 42.881950695, 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:39.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 42.881480738, 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 42.881458201999997, 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.131 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.135 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:39.137 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:39.141 INFO:tasks.rados.rados.0.smithi105.stdout:1646: done (9 left) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1648: done (8 left) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1649: done (7 left) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1650: done (6 left) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1652: done (5 left) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1653: done (4 left) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1654: done (3 left) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1655: done (2 left) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1656: done (1 left) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1657: done (0 left) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1658: tier_evict oid 40 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1658: tier_evict oid 40 snap -1 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1659: read oid 8 snap -1 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1659: expect (ObjNum 290 snap 31 seq_num 290) 2024-04-20T02:52:39.142 INFO:tasks.rados.rados.0.smithi105.stdout:1660: read oid 12 snap -1 2024-04-20T02:52:39.143 INFO:tasks.rados.rados.0.smithi105.stdout:1660: expect deleted 2024-04-20T02:52:39.143 INFO:tasks.rados.rados.0.smithi105.stdout:1661: read oid 44 snap 33 2024-04-20T02:52:39.143 INFO:tasks.rados.rados.0.smithi105.stdout:1661: expect (ObjNum 314 snap 33 seq_num 314) 2024-04-20T02:52:39.143 INFO:tasks.rados.rados.0.smithi105.stdout:1662: read oid 46 snap 33 2024-04-20T02:52:39.143 INFO:tasks.rados.rados.0.smithi105.stdout:1662: expect (ObjNum 316 snap 33 seq_num 316) 2024-04-20T02:52:39.143 INFO:tasks.rados.rados.0.smithi105.stdout:1663: tier_evict oid 7 2024-04-20T02:52:39.143 INFO:tasks.rados.rados.0.smithi105.stdout:1663: tier_evict oid 7 snap -1 2024-04-20T02:52:39.143 INFO:tasks.rados.rados.0.smithi105.stdout:1664: read oid 47 snap -1 2024-04-20T02:52:39.143 INFO:tasks.rados.rados.0.smithi105.stdout:1664: expect deleted 2024-04-20T02:52:39.143 INFO:tasks.rados.rados.0.smithi105.stdout:1665: set_chunk oid 50 target oid 2024-04-20T02:52:39.143 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:39.137+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:39.157 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:52:39.158 INFO:tasks.rados.rados.0.smithi105.stdout:1665: set_chunk oid 50 offset: 1903616 length: 483840 target oid 52f7e160d9d21ff068fe67485c2125601aece2cf1b784bd270967e1c0fae45e8 offset: 0 snap -1 2024-04-20T02:52:39.158 INFO:tasks.rados.rados.0.smithi105.stdout:1666: read oid 19 snap 34 2024-04-20T02:52:39.158 INFO:tasks.rados.rados.0.smithi105.stdout:1666: expect (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:52:39.158 INFO:tasks.rados.rados.0.smithi105.stdout:1667: set_chunk oid 22 target oid 2024-04-20T02:52:39.161 INFO:tasks.rados.rados.0.smithi105.stdout:1667: set_chunk oid 22 offset: 433664 length: 84992 target oid c88b128b0baa61aa2e105ba2d4950fee347eda35b8c96bc270b23066cf0043a8 offset: 0 snap -1 2024-04-20T02:52:39.161 INFO:tasks.rados.rados.0.smithi105.stdout:1668: read oid 4 snap 30 2024-04-20T02:52:39.161 INFO:tasks.rados.rados.0.smithi105.stdout:1668: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:39.162 INFO:tasks.rados.rados.0.smithi105.stdout:1669: write oid 6 current snap is 35 2024-04-20T02:52:39.162 INFO:tasks.rados.rados.0.smithi105.stdout:1663: got (22) Invalid argument 2024-04-20T02:52:39.162 INFO:tasks.rados.rados.0.smithi105.stdout:1669: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-04-20T02:52:39.168 INFO:tasks.rados.rados.0.smithi105.stdout:1669: writing smithi10548823-6 from 595047 to 1236802 tid 1 2024-04-20T02:52:39.173 INFO:tasks.rados.rados.0.smithi105.stdout:1669: writing smithi10548823-6 from 1667661 to 2085770 tid 2 2024-04-20T02:52:39.180 INFO:tasks.rados.rados.0.smithi105.stdout:1669: writing smithi10548823-6 from 2548663 to 3311685 tid 3 2024-04-20T02:52:39.180 INFO:tasks.rados.rados.0.smithi105.stdout:1663: done (11 left) 2024-04-20T02:52:39.180 INFO:tasks.rados.rados.0.smithi105.stdout:1670: write oid 7 current snap is 35 2024-04-20T02:52:39.181 INFO:tasks.rados.rados.0.smithi105.stdout:1658: got (22) Invalid argument 2024-04-20T02:52:39.181 INFO:tasks.rados.rados.0.smithi105.stdout:1670: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-04-20T02:52:39.188 INFO:tasks.rados.rados.0.smithi105.stdout:1670: writing smithi10548823-7 from 611854 to 1328858 tid 1 2024-04-20T02:52:39.197 INFO:tasks.rados.rados.0.smithi105.stdout:1670: writing smithi10548823-7 from 2126143 to 2687910 tid 2 2024-04-20T02:52:39.201 INFO:tasks.rados.rados.0.smithi105.stdout:1670: writing smithi10548823-7 from 3259733 to 3411854 tid 3 2024-04-20T02:52:39.201 INFO:tasks.rados.rados.0.smithi105.stdout:1658: done (11 left) 2024-04-20T02:52:39.201 INFO:tasks.rados.rados.0.smithi105.stdout:1671: read oid 28 snap -1 2024-04-20T02:52:39.201 INFO:tasks.rados.rados.0.smithi105.stdout:1671: expect (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:52:39.201 INFO:tasks.rados.rados.0.smithi105.stdout:1672: write oid 45 current snap is 35 2024-04-20T02:52:39.202 INFO:tasks.rados.rados.0.smithi105.stdout:1672: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-04-20T02:52:39.209 INFO:tasks.rados.rados.0.smithi105.stdout:1672: writing smithi10548823-45 from 628661 to 1420914 tid 1 2024-04-20T02:52:39.216 INFO:tasks.rados.rados.0.smithi105.stdout:1672: writing smithi10548823-45 from 2184625 to 2606403 tid 2 2024-04-20T02:52:39.221 INFO:tasks.rados.rados.0.smithi105.stdout:1672: writing smithi10548823-45 from 3403509 to 3428661 tid 3 2024-04-20T02:52:39.221 INFO:tasks.rados.rados.0.smithi105.stdout:1673: read oid 15 snap -1 2024-04-20T02:52:39.221 INFO:tasks.rados.rados.0.smithi105.stdout:1673: expect deleted 2024-04-20T02:52:39.221 INFO:tasks.rados.rados.0.smithi105.stdout:1674: read oid 17 snap -1 2024-04-20T02:52:39.221 INFO:tasks.rados.rados.0.smithi105.stdout:1674: expect deleted 2024-04-20T02:52:39.221 INFO:tasks.rados.rados.0.smithi105.stdout:1675: read oid 21 snap 32 2024-04-20T02:52:39.221 INFO:tasks.rados.rados.0.smithi105.stdout:1675: expect (ObjNum 300 snap 32 seq_num 300) 2024-04-20T02:52:39.222 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:39.222 INFO:tasks.rados.rados.0.smithi105.stdout:1660: done (15 left) 2024-04-20T02:52:39.222 INFO:tasks.rados.rados.0.smithi105.stdout:1676: read oid 27 snap -1 2024-04-20T02:52:39.222 INFO:tasks.rados.rados.0.smithi105.stdout:1676: expect (ObjNum 317 snap 34 seq_num 317) 2024-04-20T02:52:39.222 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:39.222 INFO:tasks.rados.rados.0.smithi105.stdout:1662: expect (ObjNum 316 snap 33 seq_num 316) 2024-04-20T02:52:39.237 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:39.238+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:39.290 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:39.290 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:39.290 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:39.290 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:39.300 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:39.338 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:39.338+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:39.439 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:39.440+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:39.455 INFO:tasks.rados.rados.0.smithi105.stdout:1659: expect (ObjNum 290 snap 31 seq_num 290) 2024-04-20T02:52:39.470 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:39.470 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:39.470 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:39.470 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:39.470 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:39.480 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:39.564 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:39.565+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:39.580 INFO:tasks.rados.rados.0.smithi105.stdout:1661: expect (ObjNum 314 snap 33 seq_num 314) 2024-04-20T02:52:39.640 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:39.641+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:39.651 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:39.651 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:39.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:39.651 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:39.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.651 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:39.651 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:39.651 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.915206398, 2024-04-20T02:52:39.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:52:39.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:52:39.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:52:39.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:52:39.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:52:39.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:52:39.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:52:39.657 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.658 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.658 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.658 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:39.658 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:39.658 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.883917337, 2024-04-20T02:52:39.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:52:39.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:52:39.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:39.664 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:39.665 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.87922688099999, 2024-04-20T02:52:39.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:52:39.665 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.665 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.665 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.665 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.665 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.666 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:52:39.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.666 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:39.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:52:39.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:39.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:52:39.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.872470412, 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.672 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:52:39.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:52:39.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.862654901, 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.678 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:52:39.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.861610969, 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.684 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:39.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.86158980899999, 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:39.690 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:39.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:52:39.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:39.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:52:39.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.654019228, 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:52:39.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.63982458300001, 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.700 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:52:39.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.633857326, 2024-04-20T02:52:39.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:52:39.708 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.62545132299999, 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.712 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.521819282, 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:52:39.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:52:39.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:52:39.716 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.291222972, 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:39.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.264278823, 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.238920556, 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.726 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:39.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.21078011199999, 2024-04-20T02:52:39.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:52:39.734 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.173411507, 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:39.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:52:39.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:52:39.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.173114584, 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.742 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.168031466, 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:52:39.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:52:39.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:52:39.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.14102890700001, 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:52:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:52:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:52:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:52:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:39.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:52:39.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:52:39.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:39.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:52:39.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:52:39.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:52:39.754 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.755 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.755 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.755 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:39.755 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:39.755 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:39.756 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:39.757 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:39.756+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:39.805 INFO:tasks.rados.rados.0.smithi105.stdout:1665: finishing set_chunk to oid 50 2024-04-20T02:52:39.805 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 89 (ObjNum 285 snap 29 seq_num 285) dirty exists 2024-04-20T02:52:39.805 INFO:tasks.rados.rados.0.smithi105.stdout:1668: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:39.857 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:39.858+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:39.928 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:39.928 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:39.928 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:39.928 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:39.938 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:39.980 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:39.980+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:40.046 INFO:tasks.rados.rados.0.smithi105.stdout:1667: finishing set_chunk to oid 22 2024-04-20T02:52:40.046 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 100 (ObjNum 292 snap 32 seq_num 292) dirty exists 2024-04-20T02:52:40.046 INFO:tasks.rados.rados.0.smithi105.stdout:1666: expect (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:52:40.058 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:40.059+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:40.107 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:40.107 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:40.107 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:40.107 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:40.107 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:40.117 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:40.141 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:52:40.193 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:40.193+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:40.203 INFO:tasks.rados.rados.0.smithi105.stdout:1671: expect (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:52:40.237 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:40.282 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:40.282+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:40.288 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:40.288 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.68429784899999, 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:52:40.289 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:40.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:52:40.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:52:40.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 118.92043145, 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:40.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:40.300 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 118.807128409, 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:68 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:40.306 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 118.801362892, 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033495717000000001, 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 68 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:40.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486849+0000", 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486859+0000", 2024-04-20T02:52:40.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.7899999999999994e-06 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486865+0000", 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8170000000000004e-06 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.486870+0000", 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7710000000000003e-06 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512015+0000", 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.025144733999999998 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.512096+0000", 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0896999999999995e-05 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520173+0000", 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0080768030000000005 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520248+0000", 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5430000000000001e-05 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.520345+0000", 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6890000000000002e-05 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:52:40.311 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 118.447342757, 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:40.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 118.43697915200001, 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.317 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:52:40.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:52:40.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:52:40.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:52:40.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:52:40.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:52:40.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:52:40.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:52:40.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:52:40.320 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 118.428894673, 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:52:40.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:52:40.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:52:40.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:52:40.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:52:40.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 100.147336448, 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:52:40.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:52:40.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:52:40.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:40.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:52:40.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:52:40.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:40.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:52:40.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:40.332 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 100.142199338, 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:52:40.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:52:40.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:52:40.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:52:40.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:52:40.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:52:40.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:52:40.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:52:40.339 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.339 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.339 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.339 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:40.339 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:40.339 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 100.141926326, 2024-04-20T02:52:40.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:52:40.340 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.340 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.340 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:40.340 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:40.340 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:40.340 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:52:40.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.341 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:40.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:40.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.342 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.342 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:52:40.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:40.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:52:40.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:52:40.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:52:40.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:52:40.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:52:40.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:52:40.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:52:40.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:52:40.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:52:40.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:52:40.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:40.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:52:40.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:52:40.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:40.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:52:40.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:52:40.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.347 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:40.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:52:40.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:52:40.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.347 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:40.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:52:40.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:52:40.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.348 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.348 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:52:40.348 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:52:40.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.348 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.348 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:52:40.349 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:52:40.349 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.349 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.349 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.349 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:40.349 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:40.349 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 100.141896941, 2024-04-20T02:52:40.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:52:40.350 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.350 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:40.350 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:40.350 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:40.350 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:40.350 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:52:40.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.351 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:40.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:40.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:40.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:40.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:52:40.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:52:40.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.354 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.354 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:52:40.354 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:52:40.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:52:40.355 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:52:40.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:52:40.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:52:40.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:40.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:52:40.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:52:40.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:52:40.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:52:40.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.358 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1675 4.3 e53/47 4:ca036f62:::smithi10548823-34:head v 53'128, mlcod=49'126)", 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.833002976000003, 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035060916999999997, 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:40.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.455209+0000", 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456603+0000", 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001394295 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456616+0000", 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2605999999999999e-05 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456631+0000", 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4735e-05 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456682+0000", 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1048000000000003e-05 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.456705+0000", 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2881e-05 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490235+0000", 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033530250999999997 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:10.490270+0000", 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5379000000000001e-05 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 77.454569641999996, 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:40.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:40.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:52:40.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:52:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:52:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:52:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:52:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.370 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2132 4.10 e70/63 4:0b0a7da5:::smithi10548823-43:head v 70'76, mlcod=65'73)", 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 72.543971603000003, 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036823724000000002, 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744239+0000", 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744631+0000", 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039175199999999999 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744644+0000", 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3079e-05 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744653+0000", 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8629999999999997e-06 2024-04-20T02:52:40.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744706+0000", 2024-04-20T02:52:40.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2627999999999998e-05 2024-04-20T02:52:40.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744737+0000", 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1387999999999998e-05 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781039+0000", 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036302147 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781064+0000", 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5182999999999999e-05 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.375 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2293 4.3 e76/60 4:c07c6daa:::smithi10548823-7:head v 76'165, mlcod=64'161)", 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.099103580000005, 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034106808000000002, 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189109+0000", 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.189108+0000", 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190758+0000", 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016504740000000001 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190778+0000", 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9507000000000001e-05 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190787+0000", 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1740000000000006e-06 2024-04-20T02:52:40.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190836+0000", 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9351999999999998e-05 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.190854+0000", 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7629e-05 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.223165+0000", 2024-04-20T02:52:40.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032310885999999997 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:34.223216+0000", 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0615000000000002e-05 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.00708942, 2024-04-20T02:52:40.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:52:40.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:52:40.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:52:40.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:52:40.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:52:40.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:52:40.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:52:40.385 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.385 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.385 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.385 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:52:40.385 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:40.385 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 51.989820985999998, 2024-04-20T02:52:40.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:52:40.386 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.386 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.386 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:40.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:40.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:52:40.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:40.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:52:40.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:52:40.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:52:40.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:52:40.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:52:40.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:52:40.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.392 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.392 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:52:40.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:52:40.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.14 98/97 [PushOp(4:2e88ea27:::smithi10548823-12:head, version: 98'33, data_included: [406769~907695,1439434~140881], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:2e88ea27:::smithi10548823-12:head@98'33, size: 2033680, copy_subset: [0~2033680], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1580315, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:40.393 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 44.038054152000001, 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033699081999999998, 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.250158+0000", 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251322+0000", 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001164251 2024-04-20T02:52:40.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251340+0000", 2024-04-20T02:52:40.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.789e-05 2024-04-20T02:52:40.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251356+0000", 2024-04-20T02:52:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5962000000000001e-05 2024-04-20T02:52:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251428+0000", 2024-04-20T02:52:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2750000000000007e-05 2024-04-20T02:52:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.251444+0000", 2024-04-20T02:52:40.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5248e-05 2024-04-20T02:52:40.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.283857+0000", 2024-04-20T02:52:40.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032413659999999997 2024-04-20T02:52:40.398 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.399 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.399 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.399 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.6.0:82 4.15 e104/103 4:a9897a5e:::smithi10548823-42:head v 104'64, mlcod=93'62)", 2024-04-20T02:52:40.399 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:40.399 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 38.152339857999998, 2024-04-20T02:52:40.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035573920000000002, 2024-04-20T02:52:40.399 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135873+0000", 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.135871+0000", 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136395+0000", 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00052370100000000001 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136409+0000", 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4191e-05 2024-04-20T02:52:40.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136418+0000", 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0100000000000001e-06 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136468+0000", 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9458e-05 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.136499+0000", 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1062999999999997e-05 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.171417+0000", 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034918607999999997 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:02.171446+0000", 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9000999999999999e-05 2024-04-20T02:52:40.403 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.816675191000002, 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:40.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:40.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:52:40.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:52:40.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:52:40.408 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.409 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.409 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.409 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:40.409 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:40.409 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:40.410 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:40.410 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:40.411 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:40.411 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:40.411+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:40.423 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:40.430 INFO:tasks.rados.rados.0.smithi105.stdout:1669: finishing write tid 1 to smithi10548823-6 2024-04-20T02:52:40.430 INFO:tasks.rados.rados.0.smithi105.stdout:1669: finishing write tid 2 to smithi10548823-6 2024-04-20T02:52:40.430 INFO:tasks.rados.rados.0.smithi105.stdout:1675: expect (ObjNum 300 snap 32 seq_num 300) 2024-04-20T02:52:40.443 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:40.462 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:40.482 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:40.502 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:40.511 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:40.511+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:40.521 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:40.522 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:40.522 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:40.522 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:40.523 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:40.523 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:40.523 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:40.523 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:40.523 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:40.535 INFO:tasks.rados.rados.0.smithi105.stdout:1669: finishing write tid 3 to smithi10548823-6 2024-04-20T02:52:40.535 INFO:tasks.rados.rados.0.smithi105.stdout:1669: finishing write tid 4 to smithi10548823-6 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1659: done (15 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1661: done (14 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1662: done (13 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1664: done (12 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1665: done (11 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1666: done (10 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1667: done (9 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1668: done (8 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1671: done (7 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1673: done (6 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1674: done (5 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1675: done (4 left) 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1677: delete oid 3 current snap is 35 2024-04-20T02:52:40.536 INFO:tasks.rados.rados.0.smithi105.stdout:1670: finishing write tid 1 to smithi10548823-7 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1669: finishing write tid 5 to smithi10548823-6 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1669: oid 6 updating version 0 to 253 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1669: oid 6 version 253 is already newer than 252 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1669: oid 6 updating version 253 to 254 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1669: oid 6 updating version 254 to 255 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 255 (ObjNum 321 snap 35 seq_num 321) dirty exists 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1669: left oid 6 (ObjNum 321 snap 35 seq_num 321) 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1672: finishing write tid 1 to smithi10548823-45 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1672: finishing write tid 2 to smithi10548823-45 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1672: finishing write tid 3 to smithi10548823-45 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1672: finishing write tid 4 to smithi10548823-45 2024-04-20T02:52:40.537 INFO:tasks.rados.rados.0.smithi105.stdout:1672: finishing write tid 5 to smithi10548823-45 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1672: oid 45 updating version 0 to 222 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1672: oid 45 version 222 is already newer than 220 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1672: oid 45 updating version 222 to 223 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1672: oid 45 version 223 is already newer than 221 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 223 (ObjNum 323 snap 35 seq_num 323) dirty exists 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1672: left oid 45 (ObjNum 323 snap 35 seq_num 323) 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1670: finishing write tid 2 to smithi10548823-7 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1670: finishing write tid 3 to smithi10548823-7 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1670: finishing write tid 4 to smithi10548823-7 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1670: finishing write tid 5 to smithi10548823-7 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1670: oid 7 updating version 0 to 225 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1670: oid 7 updating version 225 to 227 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1670: oid 7 updating version 227 to 228 2024-04-20T02:52:40.538 INFO:tasks.rados.rados.0.smithi105.stdout:1670: oid 7 version 228 is already newer than 226 2024-04-20T02:52:40.539 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 228 (ObjNum 322 snap 35 seq_num 322) dirty exists 2024-04-20T02:52:40.539 INFO:tasks.rados.rados.0.smithi105.stdout:1670: left oid 7 (ObjNum 322 snap 35 seq_num 322) 2024-04-20T02:52:40.539 INFO:tasks.rados.rados.0.smithi105.stdout:1676: expect (ObjNum 317 snap 34 seq_num 317) 2024-04-20T02:52:40.583 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:40.584 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:40.594 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:40.635 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:40.635+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:40.713 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:40.713+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:40.726 INFO:tasks.rados.rados.0.smithi105.stdout:1669: done (4 left) 2024-04-20T02:52:40.727 INFO:tasks.rados.rados.0.smithi105.stdout:1670: done (3 left) 2024-04-20T02:52:40.727 INFO:tasks.rados.rados.0.smithi105.stdout:1672: done (2 left) 2024-04-20T02:52:40.727 INFO:tasks.rados.rados.0.smithi105.stdout:1676: done (1 left) 2024-04-20T02:52:40.727 INFO:tasks.rados.rados.0.smithi105.stdout:1677: done (0 left) 2024-04-20T02:52:40.727 INFO:tasks.rados.rados.0.smithi105.stdout:1678: write oid 45 current snap is 35 2024-04-20T02:52:40.727 INFO:tasks.rados.rados.0.smithi105.stdout:1678: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2024-04-20T02:52:40.731 INFO:tasks.rados.rados.0.smithi105.stdout:1678: writing smithi10548823-45 from 662275 to 1205026 tid 1 2024-04-20T02:52:40.737 INFO:tasks.rados.rados.0.smithi105.stdout:1678: writing smithi10548823-45 from 1785236 to 2210683 tid 2 2024-04-20T02:52:40.744 INFO:tasks.rados.rados.0.smithi105.stdout:1678: writing smithi10548823-45 from 2942002 to 3462275 tid 3 2024-04-20T02:52:40.744 INFO:tasks.rados.rados.0.smithi105.stdout:1679: write oid 20 current snap is 35 2024-04-20T02:52:40.744 INFO:tasks.rados.rados.0.smithi105.stdout:1679: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-04-20T02:52:40.751 INFO:tasks.rados.rados.0.smithi105.stdout:1679: writing smithi10548823-20 from 679082 to 1297082 tid 1 2024-04-20T02:52:40.757 INFO:tasks.rados.rados.0.smithi105.stdout:1679: writing smithi10548823-20 from 1843718 to 2412823 tid 2 2024-04-20T02:52:40.763 INFO:tasks.rados.rados.0.smithi105.stdout:1679: writing smithi10548823-20 from 2853072 to 3479082 tid 3 2024-04-20T02:52:40.764 INFO:tasks.rados.rados.0.smithi105.stdout:1680: read oid 28 snap -1 2024-04-20T02:52:40.764 INFO:tasks.rados.rados.0.smithi105.stdout:1680: expect (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:52:40.764 INFO:tasks.rados.rados.0.smithi105.stdout:1681: write oid 37 current snap is 35 2024-04-20T02:52:40.764 INFO:tasks.rados.rados.0.smithi105.stdout:1681: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-04-20T02:52:40.769 INFO:tasks.rados.rados.0.smithi105.stdout:1681: writing smithi10548823-37 from 695889 to 1105491 tid 1 2024-04-20T02:52:40.772 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:40.772 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:40.776 INFO:tasks.rados.rados.0.smithi105.stdout:1681: writing smithi10548823-37 from 1618553 to 2331316 tid 2 2024-04-20T02:52:40.783 INFO:tasks.rados.rados.0.smithi105.stdout:1681: writing smithi10548823-37 from 2996848 to 3495889 tid 3 2024-04-20T02:52:40.783 INFO:tasks.rados.rados.0.smithi105.stdout:1682: set_chunk oid 29 target oid 2024-04-20T02:52:40.785 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:40.814 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:40.814+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:40.826 INFO:tasks.rados.rados.0.smithi105.stdout:1682: set_chunk oid 29 offset: 478208 length: 1468416 target oid e877726597c1de010dafcbf14ab7e25865a66f94a66bad4235fecea6883b47a4 offset: 0 snap 32 2024-04-20T02:52:40.826 INFO:tasks.rados.rados.0.smithi105.stdout:1683: read oid 19 snap -1 2024-04-20T02:52:40.826 INFO:tasks.rados.rados.0.smithi105.stdout:1683: expect (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:52:40.827 INFO:tasks.rados.rados.0.smithi105.stdout:1684: snap_remove snap 34 2024-04-20T02:52:40.915 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:40.916+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:40.916 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:40.916 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:40.927 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:41.016 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:41.017+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:41.056 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:41.057 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:41.066 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:41.117 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:41.117+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:41.202 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:41.202 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:41.212 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:41.213 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 6, 7, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2024-04-20T02:52:41.213 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:52:41.213 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-04-20T02:52:41.213 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-04-20T02:52:41.218 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:41.218+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:41.319 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:41.319+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:41.339 INFO:tasks.rados.rados.0.smithi105.stdout:1684: done (6 left) 2024-04-20T02:52:41.339 INFO:tasks.rados.rados.0.smithi105.stdout:1685: read oid 16 snap -1 2024-04-20T02:52:41.339 INFO:tasks.rados.rados.0.smithi105.stdout:1685: expect deleted 2024-04-20T02:52:41.348 INFO:tasks.rados.rados.0.smithi105.stdout:1686: set_chunk oid 17 target oid 2024-04-20T02:52:41.348 INFO:tasks.rados.rados.0.smithi105.stdout:1686: set_chunk oid 17 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:41.349 INFO:tasks.rados.rados.0.smithi105.stdout:1687: write (excl) oid 40 current snap is 35 2024-04-20T02:52:41.349 INFO:tasks.rados.rados.0.smithi105.stdout:1687: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-04-20T02:52:41.349 INFO:tasks.rados.rados.0.smithi105.stdout:1687: writing smithi10548823-40 from 712696 to 1197547 tid 1 2024-04-20T02:52:41.360 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:41.360 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:1687: writing smithi10548823-40 from 1960682 to 2533456 tid 2 2024-04-20T02:52:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:1687: writing smithi10548823-40 from 3307918 to 3512696 tid 3 2024-04-20T02:52:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:1688: read oid 36 snap -1 2024-04-20T02:52:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:1688: expect (ObjNum 304 snap 33 seq_num 304) 2024-04-20T02:52:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:1689: tier_evict oid 41 2024-04-20T02:52:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:1689: tier_evict oid 41 snap -1 2024-04-20T02:52:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:1690: write oid 46 current snap is 35 2024-04-20T02:52:41.361 INFO:tasks.rados.rados.0.smithi105.stdout:1690: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-04-20T02:52:41.372 INFO:tasks.rados.rados.0.smithi105.stdout:1690: writing smithi10548823-46 from 729503 to 1289603 tid 1 2024-04-20T02:52:41.373 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:41.378 INFO:tasks.rados.rados.0.smithi105.stdout:1690: writing smithi10548823-46 from 2019164 to 2735596 tid 2 2024-04-20T02:52:41.384 INFO:tasks.rados.rados.0.smithi105.stdout:1690: writing smithi10548823-46 from 3335341 to 3529503 tid 3 2024-04-20T02:52:41.385 INFO:tasks.rados.rados.0.smithi105.stdout:1691: read oid 22 snap -1 2024-04-20T02:52:41.385 INFO:tasks.rados.rados.0.smithi105.stdout:1691: expect (ObjNum 292 snap 32 seq_num 292) 2024-04-20T02:52:41.385 INFO:tasks.rados.rados.0.smithi105.stdout:1692: delete oid 13 current snap is 35 2024-04-20T02:52:41.385 INFO:tasks.rados.rados.0.smithi105.stdout:1680: expect (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:52:41.439 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:41.439+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:41.521 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:41.522+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:41.525 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:41.525 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:41.535 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:41.598 INFO:teuthology.orchestra.run.smithi105.stdout:moved 0 / 102 (0%) 2024-04-20T02:52:41.598 INFO:teuthology.orchestra.run.smithi105.stdout:avg 14.5714 2024-04-20T02:52:41.598 INFO:teuthology.orchestra.run.smithi105.stdout:stddev 2.32115 -> 2.32115 (expected baseline 3.53409) 2024-04-20T02:52:41.598 INFO:teuthology.orchestra.run.smithi105.stdout:min osd.5 with 11 -> 11 pgs (0.754902 -> 0.754902 * mean) 2024-04-20T02:52:41.598 INFO:teuthology.orchestra.run.smithi105.stdout:max osd.1 with 18 -> 18 pgs (1.23529 -> 1.23529 * mean) 2024-04-20T02:52:41.598 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:52:41.598 INFO:teuthology.orchestra.run.smithi105.stdout:oload 1000 2024-04-20T02:52:41.598 INFO:teuthology.orchestra.run.smithi105.stdout:max_change 0.05 2024-04-20T02:52:41.598 INFO:teuthology.orchestra.run.smithi105.stdout:max_change_osds 4 2024-04-20T02:52:41.599 INFO:teuthology.orchestra.run.smithi105.stdout:average_utilization 0.0012 2024-04-20T02:52:41.599 INFO:teuthology.orchestra.run.smithi105.stdout:overload_utilization 0.0118 2024-04-20T02:52:41.600 INFO:teuthology.orchestra.run.smithi105.stderr:no change 2024-04-20T02:52:41.613 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-04-20T02:52:41.613 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-04-20T02:52:41.626 INFO:tasks.rados.rados.0.smithi105.stdout:1678: finishing write tid 1 to smithi10548823-45 2024-04-20T02:52:41.626 INFO:tasks.rados.rados.0.smithi105.stdout:1678: finishing write tid 2 to smithi10548823-45 2024-04-20T02:52:41.626 INFO:tasks.rados.rados.0.smithi105.stdout:1678: finishing write tid 3 to smithi10548823-45 2024-04-20T02:52:41.626 INFO:tasks.rados.rados.0.smithi105.stdout:1678: finishing write tid 4 to smithi10548823-45 2024-04-20T02:52:41.626 INFO:tasks.rados.rados.0.smithi105.stdout:1678: finishing write tid 5 to smithi10548823-45 2024-04-20T02:52:41.626 INFO:tasks.rados.rados.0.smithi105.stdout:1678: oid 45 updating version 0 to 225 2024-04-20T02:52:41.626 INFO:tasks.rados.rados.0.smithi105.stdout:1678: oid 45 updating version 225 to 226 2024-04-20T02:52:41.626 INFO:tasks.rados.rados.0.smithi105.stdout:1678: oid 45 updating version 226 to 227 2024-04-20T02:52:41.626 INFO:tasks.rados.rados.0.smithi105.stdout:1678: oid 45 version 227 is already newer than 224 2024-04-20T02:52:41.626 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 227 (ObjNum 325 snap 35 seq_num 325) dirty exists 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1678: left oid 45 (ObjNum 325 snap 35 seq_num 325) 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1679: finishing write tid 1 to smithi10548823-20 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1678: done (13 left) 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1680: done (12 left) 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1692: done (11 left) 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1693: write (excl) oid 26 current snap is 35 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1679: finishing write tid 2 to smithi10548823-20 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1679: finishing write tid 3 to smithi10548823-20 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1679: finishing write tid 4 to smithi10548823-20 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1679: finishing write tid 5 to smithi10548823-20 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1679: oid 20 updating version 0 to 260 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1679: oid 20 version 260 is already newer than 258 2024-04-20T02:52:41.627 INFO:tasks.rados.rados.0.smithi105.stdout:1679: oid 20 version 260 is already newer than 257 2024-04-20T02:52:41.628 INFO:tasks.rados.rados.0.smithi105.stdout:1679: oid 20 version 260 is already newer than 259 2024-04-20T02:52:41.628 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 260 (ObjNum 326 snap 35 seq_num 326) dirty exists 2024-04-20T02:52:41.628 INFO:tasks.rados.rados.0.smithi105.stdout:1679: left oid 20 (ObjNum 326 snap 35 seq_num 326) 2024-04-20T02:52:41.628 INFO:tasks.rados.rados.0.smithi105.stdout:1681: finishing write tid 1 to smithi10548823-37 2024-04-20T02:52:41.628 INFO:tasks.rados.rados.0.smithi105.stdout:1681: finishing write tid 2 to smithi10548823-37 2024-04-20T02:52:41.628 INFO:tasks.rados.rados.0.smithi105.stdout:1681: finishing write tid 3 to smithi10548823-37 2024-04-20T02:52:41.628 INFO:tasks.rados.rados.0.smithi105.stdout:1693: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-04-20T02:52:41.628 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:41.626+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:41.633 INFO:tasks.rados.rados.0.smithi105.stdout:1693: writing smithi10548823-26 from 763117 to 1473715 tid 1 2024-04-20T02:52:41.641 INFO:tasks.rados.rados.0.smithi105.stdout:1693: writing smithi10548823-26 from 2136128 to 2856229 tid 2 2024-04-20T02:52:41.645 INFO:tasks.rados.rados.0.smithi105.stdout:1693: writing smithi10548823-26 from 3390187 to 3563117 tid 3 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:1679: done (11 left) 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:1694: tier_evict oid 11 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:1694: tier_evict oid 11 snap -1 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:1681: finishing write tid 4 to smithi10548823-37 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:1682: finishing set_chunk to oid 29 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk returned error code -95 offset: 478208 length: 1468416 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 269 snap 25 seq_num 269) dirty exists 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:1681: finishing write tid 5 to smithi10548823-37 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:1681: oid 37 updating version 0 to 90 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:1681: oid 37 updating version 90 to 91 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:1681: oid 37 version 91 is already newer than 89 2024-04-20T02:52:41.646 INFO:tasks.rados.rados.0.smithi105.stdout:1681: oid 37 updating version 91 to 92 2024-04-20T02:52:41.647 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 92 (ObjNum 327 snap 35 seq_num 327) dirty exists 2024-04-20T02:52:41.647 INFO:tasks.rados.rados.0.smithi105.stdout:1681: left oid 37 (ObjNum 327 snap 35 seq_num 327) 2024-04-20T02:52:41.647 INFO:tasks.rados.rados.0.smithi105.stdout:1683: expect (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:52:41.723 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:41.723+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:41.723 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:41.723 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:41.734 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:41.814 INFO:tasks.rados.rados.0.smithi105.stdout:1686: finishing set_chunk to oid 17 2024-04-20T02:52:41.814 INFO:tasks.rados.rados.0.smithi105.stdout:1686: got expected ENOENT (src dne) 2024-04-20T02:52:41.814 INFO:tasks.rados.rados.0.smithi105.stdout:1689: got (22) Invalid argument 2024-04-20T02:52:41.815 INFO:tasks.rados.rados.0.smithi105.stdout:1681: done (11 left) 2024-04-20T02:52:41.815 INFO:tasks.rados.rados.0.smithi105.stdout:1682: done (10 left) 2024-04-20T02:52:41.815 INFO:tasks.rados.rados.0.smithi105.stdout:1683: done (9 left) 2024-04-20T02:52:41.815 INFO:tasks.rados.rados.0.smithi105.stdout:1685: done (8 left) 2024-04-20T02:52:41.815 INFO:tasks.rados.rados.0.smithi105.stdout:1686: done (7 left) 2024-04-20T02:52:41.815 INFO:tasks.rados.rados.0.smithi105.stdout:1689: done (6 left) 2024-04-20T02:52:41.815 INFO:tasks.rados.rados.0.smithi105.stdout:1695: read oid 21 snap -1 2024-04-20T02:52:41.815 INFO:tasks.rados.rados.0.smithi105.stdout:1695: expect (ObjNum 313 snap 33 seq_num 313) 2024-04-20T02:52:41.815 INFO:tasks.rados.rados.0.smithi105.stdout:1696: set_chunk oid 24 target oid 2024-04-20T02:52:41.824 INFO:tasks.rados.rados.0.smithi105.stdout:1696: set_chunk oid 24 offset: 1556480 length: 276480 target oid 2c24ee44ba241342f39da7cde337ccf3569a54697b0f16a48d7d0e33da0f47ad offset: 0 snap 30 2024-04-20T02:52:41.824 INFO:tasks.rados.rados.0.smithi105.stdout:1697: read oid 16 snap -1 2024-04-20T02:52:41.824 INFO:tasks.rados.rados.0.smithi105.stdout:1697: expect deleted 2024-04-20T02:52:41.824 INFO:tasks.rados.rados.0.smithi105.stdout:1698: snap_create 2024-04-20T02:52:41.824 INFO:tasks.rados.rados.0.smithi105.stdout:1688: expect (ObjNum 304 snap 33 seq_num 304) 2024-04-20T02:52:41.825 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:41.825+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:41.910 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:41.910 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:41.920 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:41.962 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:41.962+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1687: finishing write tid 1 to smithi10548823-40 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1687: finishing write tid 2 to smithi10548823-40 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1687: finishing write tid 3 to smithi10548823-40 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1687: finishing write tid 4 to smithi10548823-40 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1687: finishing write tid 5 to smithi10548823-40 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1687: oid 40 updating version 0 to 378 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1687: oid 40 version 378 is already newer than 377 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1687: oid 40 updating version 378 to 379 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1687: oid 40 version 379 is already newer than 376 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 379 (ObjNum 328 snap 35 seq_num 328) dirty exists 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1687: left oid 40 (ObjNum 328 snap 35 seq_num 328) 2024-04-20T02:52:41.970 INFO:tasks.rados.rados.0.smithi105.stdout:1691: expect (ObjNum 292 snap 32 seq_num 292) 2024-04-20T02:52:42.026 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:42.026+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:42.088 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:42.089 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:42.099 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:42.126 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:42.127+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:42.152 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:52:42.198 INFO:tasks.rados.rados.0.smithi105.stdout:1690: finishing write tid 1 to smithi10548823-46 2024-04-20T02:52:42.198 INFO:tasks.rados.rados.0.smithi105.stdout:1690: finishing write tid 2 to smithi10548823-46 2024-04-20T02:52:42.198 INFO:tasks.rados.rados.0.smithi105.stdout:1690: finishing write tid 3 to smithi10548823-46 2024-04-20T02:52:42.198 INFO:tasks.rados.rados.0.smithi105.stdout:1690: finishing write tid 4 to smithi10548823-46 2024-04-20T02:52:42.198 INFO:tasks.rados.rados.0.smithi105.stdout:1690: finishing write tid 5 to smithi10548823-46 2024-04-20T02:52:42.198 INFO:tasks.rados.rados.0.smithi105.stdout:1690: oid 46 updating version 0 to 98 2024-04-20T02:52:42.198 INFO:tasks.rados.rados.0.smithi105.stdout:1690: oid 46 version 98 is already newer than 97 2024-04-20T02:52:42.198 INFO:tasks.rados.rados.0.smithi105.stdout:1690: oid 46 updating version 98 to 100 2024-04-20T02:52:42.198 INFO:tasks.rados.rados.0.smithi105.stdout:1690: oid 46 version 100 is already newer than 99 2024-04-20T02:52:42.198 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 100 (ObjNum 329 snap 35 seq_num 329) dirty exists 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1690: left oid 46 (ObjNum 329 snap 35 seq_num 329) 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1694: got (22) Invalid argument 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1693: finishing write tid 1 to smithi10548823-26 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1693: finishing write tid 2 to smithi10548823-26 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1693: finishing write tid 3 to smithi10548823-26 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1693: finishing write tid 4 to smithi10548823-26 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1693: finishing write tid 5 to smithi10548823-26 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1693: oid 26 updating version 0 to 384 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1693: oid 26 version 384 is already newer than 383 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1693: oid 26 updating version 384 to 385 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1693: oid 26 updating version 385 to 386 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 386 (ObjNum 331 snap 35 seq_num 331) dirty exists 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1693: left oid 26 (ObjNum 331 snap 35 seq_num 331) 2024-04-20T02:52:42.199 INFO:tasks.rados.rados.0.smithi105.stdout:1695: expect (ObjNum 313 snap 33 seq_num 313) 2024-04-20T02:52:42.228 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:42.228+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:42.273 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:42.273 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:42.283 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout:moved 9 / 102 (8.82353%) 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout:avg 14.5714 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout:stddev 2.32115 -> 1.4983 (expected baseline 3.53409) 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout:min osd.5 with 11 -> 13 pgs (0.754902 -> 0.892157 * mean) 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout:max osd.1 with 18 -> 16 pgs (1.23529 -> 1.09804 * mean) 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout:oload 110 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout:max_change 3 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout:max_change_osds 4 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout:average_utilization 0.0012 2024-04-20T02:52:42.353 INFO:teuthology.orchestra.run.smithi105.stdout:overload_utilization 0.0013 2024-04-20T02:52:42.354 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4 weight 1.0000 -> 0.7572 2024-04-20T02:52:42.354 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1 weight 1.0000 -> 0.8222 2024-04-20T02:52:42.354 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0 weight 1.0000 -> 0.8317 2024-04-20T02:52:42.354 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:42.353+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:42.359 INFO:tasks.rados.rados.0.smithi105.stdout:1696: finishing set_chunk to oid 24 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1687: done (9 left) 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1688: done (8 left) 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1690: done (7 left) 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1691: done (6 left) 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1693: done (5 left) 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1694: done (4 left) 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1695: done (3 left) 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1696: done (2 left) 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1697: done (1 left) 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1698: done (0 left) 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1699: read oid 32 snap -1 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1699: expect deleted 2024-04-20T02:52:42.360 INFO:tasks.rados.rados.0.smithi105.stdout:1700: read oid 35 snap -1 2024-04-20T02:52:42.361 INFO:tasks.rados.rados.0.smithi105.stdout:1700: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:52:42.361 INFO:tasks.rados.rados.0.smithi105.stdout:1701: read oid 37 snap -1 2024-04-20T02:52:42.361 INFO:tasks.rados.rados.0.smithi105.stdout:1701: expect (ObjNum 327 snap 35 seq_num 327) 2024-04-20T02:52:42.361 INFO:tasks.rados.rados.0.smithi105.stdout:1702: write (excl) oid 14 current snap is 36 2024-04-20T02:52:42.361 INFO:tasks.rados.rados.0.smithi105.stdout:1702: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-04-20T02:52:42.369 INFO:tasks.rados.rados.0.smithi105.stdout:1702: writing smithi10548823-14 from 779924 to 1565771 tid 1 2024-04-20T02:52:42.376 INFO:tasks.rados.rados.0.smithi105.stdout:1702: writing smithi10548823-14 from 2078257 to 2658369 tid 2 2024-04-20T02:52:42.378 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-04-20T02:52:42.378 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2024-04-20T02:52:42.381 INFO:tasks.rados.rados.0.smithi105.stdout:1702: writing smithi10548823-14 from 3301257 to 3579924 tid 3 2024-04-20T02:52:42.381 INFO:tasks.rados.rados.0.smithi105.stdout:1703: set_chunk oid 36 target oid 2024-04-20T02:52:42.430 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:42.430+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:42.470 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:42.470 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:42.481 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:42.531 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:42.531+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:42.534 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:42.620 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:42.620 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:42.631 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:42.632 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:42.632+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:42.704 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:42.725 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:42.732 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:42.732+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:42.736 INFO:teuthology.orchestra.run.smithi105.stdout:moved 0 / 102 (0%) 2024-04-20T02:52:42.736 INFO:teuthology.orchestra.run.smithi105.stdout:avg 14.5714 2024-04-20T02:52:42.736 INFO:teuthology.orchestra.run.smithi105.stdout:stddev 1.4983 -> 1.4983 (expected baseline 3.53409) 2024-04-20T02:52:42.736 INFO:teuthology.orchestra.run.smithi105.stdout:min osd.3 with 13 -> 13 pgs (0.892157 -> 0.892157 * mean) 2024-04-20T02:52:42.736 INFO:teuthology.orchestra.run.smithi105.stdout:max osd.6 with 17 -> 17 pgs (1.16667 -> 1.16667 * mean) 2024-04-20T02:52:42.736 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:52:42.737 INFO:teuthology.orchestra.run.smithi105.stdout:oload 1000 2024-04-20T02:52:42.737 INFO:teuthology.orchestra.run.smithi105.stdout:max_change 0.05 2024-04-20T02:52:42.737 INFO:teuthology.orchestra.run.smithi105.stdout:max_change_osds 4 2024-04-20T02:52:42.737 INFO:teuthology.orchestra.run.smithi105.stdout:average_utilization 0.0012 2024-04-20T02:52:42.737 INFO:teuthology.orchestra.run.smithi105.stdout:overload_utilization 0.0118 2024-04-20T02:52:42.738 INFO:teuthology.orchestra.run.smithi105.stderr:no change 2024-04-20T02:52:42.745 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:42.750 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-04-20T02:52:42.751 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2024-04-20T02:52:42.765 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:42.780 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:42.780 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:42.780 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:42.781 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:42.781 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:42.788 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:42.793 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:42.809 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:42.829 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:42.833 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:42.833+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:42.850 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:42.851 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:42.852 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:42.852 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:42.852 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:42.852 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:42.852 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:42.852 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:42.852 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:42.934 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:42.934+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:42.957 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:42.957 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:42.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 116.412768568, 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:52:42.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:52:42.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:42.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:52:42.963 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 116.40654749399999, 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:52:42.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:52:42.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:42.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:42.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:970 4.0 4:007b2a87:::smithi10548823-39:head [write 2687912~589420] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 116.39814729, 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051519626999999998, 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 970 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558361+0000", 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558358+0000", 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:42.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558645+0000", 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028712200000000001 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558661+0000", 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5971999999999999e-05 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558672+0000", 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0798000000000001e-05 2024-04-20T02:52:42.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579158+0000", 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020486239999999999 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579288+0000", 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130407 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.579461+0000", 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017325500000000001 2024-04-20T02:52:42.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594660+0000", 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015198208 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595715+0000", 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0010555709999999999 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609792+0000", 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014077058 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609840+0000", 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7615999999999999e-05 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.609880+0000", 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0473999999999999e-05 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.975 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:972 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 116.39780464499999, 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.051533084, 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 972 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.976 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558703+0000", 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:42.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558705+0000", 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9300000000000002e-06 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558711+0000", 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9290000000000003e-06 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:42.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.558714+0000", 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.084e-06 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595221+0000", 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036507481000000001 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595300+0000", 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8615000000000005e-05 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610043+0000", 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014742936999999999 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610138+0000", 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4988000000000006e-05 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.610237+0000", 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8519999999999999e-05 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.980 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 108.591482642, 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.981 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:52:42.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:42.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.986 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 108.58421968, 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.987 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:52:42.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:52:42.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:42.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:42.992 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 108.57827715099999, 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:42.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:52:42.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:52:42.998 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 108.578074159, 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:52:42.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.004 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 108.57805123200001, 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:52:43.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:52:43.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 108.567967498, 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:52:43.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:52:43.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:52:43.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 108.55108208999999, 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053383991999999998, 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405426+0000", 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.405424+0000", 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415508+0000", 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010084081 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415522+0000", 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.34e-05 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415531+0000", 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1800000000000002e-06 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415594+0000", 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3436999999999996e-05 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.415621+0000", 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7195e-05 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458788+0000", 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043166983999999999 2024-04-20T02:52:43.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458810+0000", 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1548000000000001e-05 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 108.53480564900001, 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053922266000000003, 2024-04-20T02:52:43.018 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421702+0000", 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.421701+0000", 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426938+0000", 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0052371659999999997 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426950+0000", 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2159e-05 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.426958+0000", 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5809999999999999e-06 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431015+0000", 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0040571610000000001 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431050+0000", 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4932999999999999e-05 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475585+0000", 2024-04-20T02:52:43.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.044535309000000002 2024-04-20T02:52:43.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.475625+0000", 2024-04-20T02:52:43.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9715999999999997e-05 2024-04-20T02:52:43.022 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.022 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.022 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 93.510250177000003, 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.026 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:43.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:43.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:52:43.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:52:43.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:43.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:43.032 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 78.195934401000002, 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:52:43.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2545 4.9 e91/81 4:95e19a88:::smithi10548823-40:head v 91'253, mlcod=91'251)", 2024-04-20T02:52:43.038 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 54.541297286000002, 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054757404000000003, 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415211+0000", 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.415210+0000", 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416004+0000", 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00079381199999999997 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416022+0000", 2024-04-20T02:52:43.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7969e-05 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416031+0000", 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9279999999999999e-06 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416100+0000", 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8891000000000002e-05 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.416128+0000", 2024-04-20T02:52:43.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8203000000000001e-05 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469912+0000", 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053783794000000003 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:48.469968+0000", 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6474000000000002e-05 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.608268479000003, 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:52:43.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:52:43.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2700 4.0 4:007b2a87:::smithi10548823-39:head [write 1967200~714747] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.599778284999999, 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061846287, 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2700 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356728+0000", 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356879+0000", 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000150958 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356894+0000", 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5132e-05 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356906+0000", 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2071e-05 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366666+0000", 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0097595289999999994 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366774+0000", 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108527 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366956+0000", 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181359 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390515+0000", 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023559902000000001 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406790+0000", 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016274555999999999 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418488+0000", 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011697591 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418529+0000", 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1545000000000003e-05 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418576+0000", 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6903000000000001e-05 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.054 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2701 4.0 4:007b2a87:::smithi10548823-39:head [write 3421983~426820] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.592602292999999, 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054799176999999998, 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2701 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363904+0000", 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364154+0000", 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000250179 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364171+0000", 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6748e-05 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364182+0000", 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1371e-05 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390552+0000", 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026369607 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390652+0000", 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010059499999999999 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390857+0000", 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020445199999999999 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418224+0000", 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027367316999999999 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418282+0000", 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8279000000000002e-05 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418618+0000", 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000335967 2024-04-20T02:52:43.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418661+0000", 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2614999999999998e-05 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418705+0000", 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3993999999999999e-05 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2702 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (57) in=7b,truncate 3848803] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.592316238999999, 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058669165000000002, 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2702 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364191+0000", 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364196+0000", 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4490000000000001e-06 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364201+0000", 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5040000000000002e-06 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364206+0000", 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:52:43.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.405895+0000", 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041689047 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406021+0000", 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000125561 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406541+0000", 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052000699999999996 2024-04-20T02:52:43.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418451+0000", 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011910041999999999 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418747+0000", 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029622399999999999 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422704+0000", 2024-04-20T02:52:43.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039571859999999997 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422781+0000", 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.661e-05 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422861+0000", 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9987999999999994e-05 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2703 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 50.592297187, 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058940278999999998, 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2703 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:43.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364213+0000", 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4830000000000002e-06 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364220+0000", 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.007e-06 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364225+0000", 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0350000000000001e-06 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406820+0000", 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042595135999999999 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406923+0000", 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102846 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422897+0000", 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015974202999999999 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423015+0000", 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117581 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423151+0000", 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013635899999999999 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:43.071 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:43.073 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:43.073 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:43.073+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:43.117 INFO:teuthology.orchestra.run.smithi105.stdout:moved 0 / 102 (0%) 2024-04-20T02:52:43.118 INFO:teuthology.orchestra.run.smithi105.stdout:avg 14.5714 2024-04-20T02:52:43.118 INFO:teuthology.orchestra.run.smithi105.stdout:stddev 1.4983 -> 1.4983 (expected baseline 3.53409) 2024-04-20T02:52:43.118 INFO:teuthology.orchestra.run.smithi105.stdout:min osd.3 with 13 -> 13 pgs (0.892157 -> 0.892157 * mean) 2024-04-20T02:52:43.118 INFO:teuthology.orchestra.run.smithi105.stdout:max osd.6 with 17 -> 17 pgs (1.16667 -> 1.16667 * mean) 2024-04-20T02:52:43.118 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:52:43.118 INFO:teuthology.orchestra.run.smithi105.stdout:oload 1000 2024-04-20T02:52:43.118 INFO:teuthology.orchestra.run.smithi105.stdout:max_change 0.05 2024-04-20T02:52:43.118 INFO:teuthology.orchestra.run.smithi105.stdout:max_change_osds 4 2024-04-20T02:52:43.118 INFO:teuthology.orchestra.run.smithi105.stdout:average_utilization 0.0012 2024-04-20T02:52:43.118 INFO:teuthology.orchestra.run.smithi105.stdout:overload_utilization 0.0118 2024-04-20T02:52:43.120 INFO:teuthology.orchestra.run.smithi105.stderr:no change 2024-04-20T02:52:43.131 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-04-20T02:52:43.132 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-04-20T02:52:43.175 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:43.175+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:43.214 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:43.215 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:43.215 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:43.215 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:43.225 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:43.274 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:43.275+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:43.354 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:52:43.358 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:43.358 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:43.358 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:43.358 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:43.359 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:43.369 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:43.409 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:43.409+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:43.466 INFO:tasks.ceph.mgr.x.smithi138.stderr:2024-04-20T02:52:43.466+0000 7f605a8e1700 -1 mgr.server reply reply (3) No such process no change 2024-04-20T02:52:43.466 INFO:teuthology.orchestra.run.smithi105.stdout:moved 12 / 102 (11.7647%) 2024-04-20T02:52:43.466 INFO:teuthology.orchestra.run.smithi105.stdout:avg 14.5714 2024-04-20T02:52:43.466 INFO:teuthology.orchestra.run.smithi105.stdout:stddev 1.4983 -> 2.96923 (expected baseline 3.53409) 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout:min osd.3 with 13 -> 16 pgs (0.892157 -> 1.09804 * mean) 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout:max osd.6 with 17 -> 17 pgs (1.16667 -> 1.16667 * mean) 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout:oload 110 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout:max_change 1 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout:max_change_osds 4 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout:average_utilization 0.0012 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout:overload_utilization 0.0013 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4 weight 0.7572 -> 0.5806 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1 weight 0.8222 -> 0.6326 2024-04-20T02:52:43.467 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0 weight 0.8317 -> 0.7004 2024-04-20T02:52:43.469 INFO:teuthology.orchestra.run.smithi105.stderr:no change 2024-04-20T02:52:43.475 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:43.475+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:43.541 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:43.541 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:43.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.597593936, 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:52:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.547 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.590115618, 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.548 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:43.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:52:43.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:52:43.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:52:43.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.585357719, 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.558 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 109.161928785, 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:43.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:52:43.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:52:43.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:52:43.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:52:43.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:52:43.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 109.16189190199999, 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.565 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:43.571 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 109.161867269, 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:52:43.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:52:43.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:52:43.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 109.155373858, 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.576 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:43.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:52:43.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:52:43.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:43.583 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 109.149309836, 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:43.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:52:43.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:52:43.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:52:43.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:52:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 109.139248071, 2024-04-20T02:52:43.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:52:43.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 109.138555467, 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.595 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:43.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:43.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:43.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:52:43.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:52:43.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:52:43.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.601 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 109.138543813, 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:43.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:52:43.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:52:43.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:52:43.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 103.432479257, 2024-04-20T02:52:43.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:52:43.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:52:43.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 103.413418541, 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:52:43.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:52:43.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:52:43.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 103.407472227, 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:43.622 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 103.40690678599999, 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:52:43.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:52:43.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:52:43.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:52:43.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 103.406880755, 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:52:43.628 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 47.404667953999997, 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.633 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:43.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:52:43.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:52:43.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:52:43.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:52:43.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 47.399318430000001, 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.640 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:52:43.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:43.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.644 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 47.398848473000001, 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:52:43.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:52:43.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 47.398825936999998, 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:52:43.649 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.654 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.654 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:43.654 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:43.654 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:43.655 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:43.656 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:43.656+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:43.756 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:43.757+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:43.784 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:43.784 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:43.784 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:43.784 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:43.794 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:43.857 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:43.858+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:43.958 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:43.958+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:43.964 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:43.964 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:43.964 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:43.964 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:43.964 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:43.975 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:44.059 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:44.060+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:44.145 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.409713547, 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.146 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:52:44.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:52:44.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:52:44.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.378424486, 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.152 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:52:44.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:44.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.37373402999999, 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:52:44.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.366977561, 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:44.169 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.35716205, 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:52:44.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:52:44.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:52:44.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.356118118, 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.176 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.181 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.35609695799999, 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:52:44.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:44.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:52:44.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:52:44.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.187 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:44.187 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:44.187 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.148526377, 2024-04-20T02:52:44.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:52:44.187 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.187 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.187 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.13433173200001, 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.191 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:52:44.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:52:44.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.128364475, 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:52:44.195 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:44.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:52:44.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:52:44.200 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.200 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.200 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.200 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:52:44.200 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.11995847199999, 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:52:44.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.016326431, 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.208 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 121.785730121, 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:52:44.212 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 121.758785972, 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:52:44.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:52:44.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:52:44.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 121.733427705, 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 121.705287261, 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 121.667918656, 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.225 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:52:44.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:52:44.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:52:44.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 121.667621733, 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:52:44.229 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:52:44.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:52:44.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 121.662538615, 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:52:44.233 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:52:44.237 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 121.63553605600001, 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:44.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:52:44.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:52:44.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.241 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:44.243 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:44.243 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:44.243+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:44.344 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:44.344+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:44.398 INFO:tasks.rados.rados.0.smithi105.stdout:1703: set_chunk oid 36 offset: 1748992 length: 1131520 target oid 816110ba719f6890566522e3b5463f2d394b1d6ac9be449e15451e891fdd141f offset: 0 snap -1 2024-04-20T02:52:44.398 INFO:tasks.rados.rados.0.smithi105.stdout:1704: write oid 41 current snap is 36 2024-04-20T02:52:44.398 INFO:tasks.rados.rados.0.smithi105.stdout:1704: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-04-20T02:52:44.404 INFO:tasks.rados.rados.0.smithi105.stdout:1704: writing smithi10548823-41 from 796731 to 1257827 tid 1 2024-04-20T02:52:44.412 INFO:tasks.rados.rados.0.smithi105.stdout:1704: writing smithi10548823-41 from 1736739 to 2460509 tid 2 2024-04-20T02:52:44.418 INFO:tasks.rados.rados.0.smithi105.stdout:1704: writing smithi10548823-41 from 2928680 to 3596731 tid 3 2024-04-20T02:52:44.418 INFO:tasks.rados.rados.0.smithi105.stdout:1705: read oid 4 snap -1 2024-04-20T02:52:44.418 INFO:tasks.rados.rados.0.smithi105.stdout:1705: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:44.418 INFO:tasks.rados.rados.0.smithi105.stdout:1706: write (excl) oid 46 current snap is 36 2024-04-20T02:52:44.419 INFO:tasks.rados.rados.0.smithi105.stdout:1706: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-04-20T02:52:44.423 INFO:tasks.rados.rados.0.smithi105.stdout:1706: writing smithi10548823-46 from 413538 to 949883 tid 1 2024-04-20T02:52:44.430 INFO:tasks.rados.rados.0.smithi105.stdout:1706: writing smithi10548823-46 from 1395221 to 1979002 tid 2 2024-04-20T02:52:44.436 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:44.436 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:44.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.436 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3880 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'231, mlcod=141'228)", 2024-04-20T02:52:44.436 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.412474+0000", 2024-04-20T02:52:44.436 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.024857166, 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024881944999999999, 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.412474+0000", 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.412474+0000", 2024-04-20T02:52:44.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.412473+0000", 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.413004+0000", 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053028500000000002 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.413016+0000", 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2095e-05 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.413023+0000", 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3529999999999997e-06 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.413070+0000", 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7481999999999997e-05 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.413098+0000", 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7291999999999999e-05 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3881 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'232, mlcod=141'228)", 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.018671084000000001, 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018768173999999999, 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:44.440 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418659+0000", 2024-04-20T02:52:44.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419496+0000", 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00083715799999999996 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419513+0000", 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7489000000000001e-05 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419524+0000", 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0118e-05 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429881+0000", 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010357762 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429905+0000", 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3546000000000001e-05 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.443 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.444 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.444 INFO:teuthology.orchestra.run.smithi138.stdout: ], 2024-04-20T02:52:44.444 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 2 2024-04-20T02:52:44.444 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:44.445 INFO:tasks.rados.rados.0.smithi105.stdout:1706: writing smithi10548823-46 from 2556103 to 3214720 tid 3 2024-04-20T02:52:44.445 INFO:tasks.rados.rados.0.smithi105.stdout:1707: write (excl) oid 13 current snap is 36 2024-04-20T02:52:44.445 INFO:tasks.rados.rados.0.smithi105.stdout:1707: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-04-20T02:52:44.445 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:44.445+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:44.446 INFO:tasks.rados.rados.0.smithi105.stdout:1707: writing smithi10548823-13 from 430345 to 1158292 tid 1 2024-04-20T02:52:44.453 INFO:tasks.rados.rados.0.smithi105.stdout:1707: writing smithi10548823-13 from 1570056 to 2297495 tid 2 2024-04-20T02:52:44.458 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:44.460 INFO:tasks.rados.rados.0.smithi105.stdout:1707: writing smithi10548823-13 from 2699879 to 3169768 tid 3 2024-04-20T02:52:44.461 INFO:tasks.rados.rados.0.smithi105.stdout:1708: write oid 8 current snap is 36 2024-04-20T02:52:44.461 INFO:tasks.rados.rados.0.smithi105.stdout:1708: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-04-20T02:52:44.464 INFO:tasks.rados.rados.0.smithi105.stdout:1708: writing smithi10548823-8 from 447152 to 850348 tid 1 2024-04-20T02:52:44.475 INFO:tasks.rados.rados.0.smithi105.stdout:1708: writing smithi10548823-8 from 1512185 to 2099635 tid 2 2024-04-20T02:52:44.480 INFO:tasks.rados.rados.0.smithi105.stdout:1708: writing smithi10548823-8 from 2727302 to 3408463 tid 3 2024-04-20T02:52:44.485 INFO:tasks.rados.rados.0.smithi105.stdout:1709: delete oid 24 current snap is 36 2024-04-20T02:52:44.485 INFO:tasks.rados.rados.0.smithi105.stdout:1700: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:52:44.546 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:44.546+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:44.567 INFO:tasks.rados.rados.0.smithi105.stdout:1699: done (10 left) 2024-04-20T02:52:44.567 INFO:tasks.rados.rados.0.smithi105.stdout:1700: done (9 left) 2024-04-20T02:52:44.568 INFO:tasks.rados.rados.0.smithi105.stdout:1709: done (8 left) 2024-04-20T02:52:44.568 INFO:tasks.rados.rados.0.smithi105.stdout:1710: delete oid 17 current snap is 36 2024-04-20T02:52:44.568 INFO:tasks.rados.rados.0.smithi105.stdout:1701: expect (ObjNum 327 snap 35 seq_num 327) 2024-04-20T02:52:44.621 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:44.622 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:44.622 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:44.622 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:44.622 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:44.636 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:44.647 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:44.647+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:44.748 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:44.749+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:44.774 INFO:tasks.rados.rados.0.smithi105.stdout:1703: finishing set_chunk to oid 36 2024-04-20T02:52:44.775 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 337 (ObjNum 304 snap 33 seq_num 304) dirty exists 2024-04-20T02:52:44.775 INFO:tasks.rados.rados.0.smithi105.stdout:1702: finishing write tid 1 to smithi10548823-14 2024-04-20T02:52:44.775 INFO:tasks.rados.rados.0.smithi105.stdout:1705: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:44.814 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:44.814 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.210068237, 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:52:44.815 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:52:44.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:52:44.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:52:44.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.821 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:25 4.3 e29/27 4:c07c6daa:::smithi10548823-7:head v 29'7, mlcod=28'3)", 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 123.44620183799999, 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036863185999999999, 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367781+0000", 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.367780+0000", 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369163+0000", 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001382435 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369178+0000", 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5590000000000002e-05 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369187+0000", 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7309999999999999e-06 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.824 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369289+0000", 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010160199999999999 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.369313+0000", 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4009e-05 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404598+0000", 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.035285182999999998 2024-04-20T02:52:44.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.404644+0000", 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6409000000000002e-05 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 123.332898797, 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:52:44.826 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.973113145, 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.832 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:52:44.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:52:44.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:52:44.836 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.96274954, 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:44.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.954665061, 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:52:44.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:52:44.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:52:44.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:52:44.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:52:44.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.845 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.673106836, 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.846 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.851 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.667969726, 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:52:44.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:52:44.857 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.667696714, 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.858 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:52:44.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:52:44.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.863 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.667667329, 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.864 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:52:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 81.980340029999994, 2024-04-20T02:52:44.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:52:44.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.876 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2132 4.10 e70/63 4:0b0a7da5:::smithi10548823-43:head v 70'76, mlcod=65'73)", 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 77.069741991000001, 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036823724000000002, 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744241+0000", 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744239+0000", 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744631+0000", 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039175199999999999 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744644+0000", 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3079e-05 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744653+0000", 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8629999999999997e-06 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744706+0000", 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2627999999999998e-05 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.744737+0000", 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1387999999999998e-05 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781039+0000", 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036302147 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:27.781064+0000", 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5182999999999999e-05 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.880 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 56.532859807999998, 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:44.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:52:44.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:52:44.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 56.515591374000003, 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:52:44.885 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:52:44.889 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.342445579, 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:44.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:52:44.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:52:44.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3881 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'232, mlcod=141'228)", 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.39532303899999999, 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050172267999999999, 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418659+0000", 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419496+0000", 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00083715799999999996 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419513+0000", 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7489000000000001e-05 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419524+0000", 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0118e-05 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429881+0000", 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010357762 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429905+0000", 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3546000000000001e-05 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468790+0000", 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038884821 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468832+0000", 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2341e-05 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.897 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3898 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'133, mlcod=133'129)", 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.34316406700000002, 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038834664999999997, 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470817+0000", 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471051+0000", 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023384600000000001 2024-04-20T02:52:44.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471066+0000", 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4205e-05 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471074+0000", 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4090000000000003e-06 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472816+0000", 2024-04-20T02:52:44.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017419429999999999 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472843+0000", 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7296000000000001e-05 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509592+0000", 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036748496999999998 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509653+0000", 2024-04-20T02:52:44.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1707999999999999e-05 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3899 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'134, mlcod=133'129)", 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.32589763100000002, 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038218993999999999, 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488084+0000", 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488262+0000", 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017768600000000001 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488274+0000", 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2388000000000001e-05 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488283+0000", 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3840000000000003e-06 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489094+0000", 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081045100000000001 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489119+0000", 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4751000000000001e-05 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526264+0000", 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037144912000000002 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526304+0000", 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0513000000000003e-05 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3891 4.7 e146/145 4:e2c0d65b:::smithi10548823-46:head v 146'105, mlcod=143'100)", 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.32491753600000001, 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041132308999999999, 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:44.906 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:44.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489072+0000", 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9539999999999998e-06 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489082+0000", 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0329000000000001e-05 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489089+0000", 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4959999999999999e-06 2024-04-20T02:52:44.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489814+0000", 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072423699999999997 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489837+0000", 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3218999999999999e-05 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530157+0000", 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040319762000000002 2024-04-20T02:52:44.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530198+0000", 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0886000000000001e-05 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.7 146/145 [PushOp(4:e2c0d65b:::smithi10548823-46:43, version: 143'96, data_included: [426944~820638,1468778~227938], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:e2c0d65b:::smithi10548823-46:43@143'96, size: 3311012, copy_subset: [0~3311012], clone_subset: {}, snapset: 45=[]:{3f=[3f,39],43=[41],45=[45]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1696716, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.32370618400000001, 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039876652999999998, 2024-04-20T02:52:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490276+0000", 2024-04-20T02:52:44.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490785+0000", 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050941300000000001 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490804+0000", 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9080999999999999e-05 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:44.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490823+0000", 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.855e-05 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498138+0000", 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007315004 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498153+0000", 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5515e-05 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530153+0000", 2024-04-20T02:52:44.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031999845999999998 2024-04-20T02:52:44.914 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.914 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.914 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.914 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:44.914 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:44.914 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:44.915 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:44.916 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:44.917 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:44.916+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:45.016 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:45.017+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:45.047 INFO:tasks.rados.rados.0.smithi105.stdout:1702: finishing write tid 2 to smithi10548823-14 2024-04-20T02:52:45.047 INFO:tasks.rados.rados.0.smithi105.stdout:1702: finishing write tid 3 to smithi10548823-14 2024-04-20T02:52:45.047 INFO:tasks.rados.rados.0.smithi105.stdout:1702: finishing write tid 4 to smithi10548823-14 2024-04-20T02:52:45.047 INFO:tasks.rados.rados.0.smithi105.stdout:1702: finishing write tid 5 to smithi10548823-14 2024-04-20T02:52:45.047 INFO:tasks.rados.rados.0.smithi105.stdout:1702: oid 14 updating version 0 to 119 2024-04-20T02:52:45.047 INFO:tasks.rados.rados.0.smithi105.stdout:1702: oid 14 updating version 119 to 120 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1702: oid 14 updating version 120 to 121 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1702: oid 14 version 121 is already newer than 118 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 121 (ObjNum 332 snap 36 seq_num 332) dirty exists 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1702: left oid 14 (ObjNum 332 snap 36 seq_num 332) 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1704: finishing write tid 1 to smithi10548823-41 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1704: finishing write tid 2 to smithi10548823-41 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1704: finishing write tid 3 to smithi10548823-41 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1704: finishing write tid 4 to smithi10548823-41 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1701: done (8 left) 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1702: done (7 left) 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1703: done (6 left) 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1705: done (5 left) 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1710: done (4 left) 2024-04-20T02:52:45.048 INFO:tasks.rados.rados.0.smithi105.stdout:1711: snap_remove snap 30 2024-04-20T02:52:45.070 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:45.091 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:45.111 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:45.115 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:45.115 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:45.115 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:45.115 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:45.117 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:45.118+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:45.125 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:45.132 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:45.153 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:45.174 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:45.195 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:45.216 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:45.217 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:45.217 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:45.218 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:45.218 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:45.218 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:45.218 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:45.218 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:45.218 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:45.218 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:45.218+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:45.297 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:45.297 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:45.307 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:45.318 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:45.319+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:45.368 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:52:45.372 INFO:tasks.rados.rados.0.smithi105.stdout:1711: done (4 left) 2024-04-20T02:52:45.372 INFO:tasks.rados.rados.0.smithi105.stdout:1712: read oid 31 snap -1 2024-04-20T02:52:45.372 INFO:tasks.rados.rados.0.smithi105.stdout:1712: expect (ObjNum 305 snap 33 seq_num 305) 2024-04-20T02:52:45.372 INFO:tasks.rados.rados.0.smithi105.stdout:1713: write (excl) oid 5 current snap is 36 2024-04-20T02:52:45.372 INFO:tasks.rados.rados.0.smithi105.stdout:1713: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-04-20T02:52:45.378 INFO:tasks.rados.rados.0.smithi105.stdout:1713: writing smithi10548823-5 from 497573 to 1126516 tid 1 2024-04-20T02:52:45.387 INFO:tasks.rados.rados.0.smithi105.stdout:1713: writing smithi10548823-5 from 1687631 to 2422408 tid 2 2024-04-20T02:52:45.396 INFO:tasks.rados.rados.0.smithi105.stdout:1713: writing smithi10548823-5 from 3093218 to 3697573 tid 3 2024-04-20T02:52:45.396 INFO:tasks.rados.rados.0.smithi105.stdout:1714: tier_evict oid 30 2024-04-20T02:52:45.396 INFO:tasks.rados.rados.0.smithi105.stdout:1714: tier_evict oid 30 snap -1 2024-04-20T02:52:45.396 INFO:tasks.rados.rados.0.smithi105.stdout:1704: finishing write tid 5 to smithi10548823-41 2024-04-20T02:52:45.396 INFO:tasks.rados.rados.0.smithi105.stdout:1704: oid 41 updating version 0 to 231 2024-04-20T02:52:45.396 INFO:tasks.rados.rados.0.smithi105.stdout:1704: oid 41 updating version 231 to 233 2024-04-20T02:52:45.396 INFO:tasks.rados.rados.0.smithi105.stdout:1704: oid 41 version 233 is already newer than 232 2024-04-20T02:52:45.396 INFO:tasks.rados.rados.0.smithi105.stdout:1704: oid 41 updating version 233 to 234 2024-04-20T02:52:45.396 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 234 (ObjNum 333 snap 36 seq_num 333) dirty exists 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1704: left oid 41 (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1706: finishing write tid 1 to smithi10548823-46 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1706: finishing write tid 2 to smithi10548823-46 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1704: done (6 left) 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1715: read oid 11 snap -1 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1715: expect (ObjNum 282 snap 27 seq_num 282) 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1716: read oid 24 snap -1 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1716: expect deleted 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1717: read oid 2 snap -1 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1717: expect (ObjNum 307 snap 33 seq_num 307) 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1718: read oid 4 snap -1 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1718: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1719: set_chunk oid 10 target oid 2024-04-20T02:52:45.397 INFO:tasks.rados.rados.0.smithi105.stdout:1719: set_chunk oid 10 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:45.398 INFO:tasks.rados.rados.0.smithi105.stdout:1720: read oid 1 snap -1 2024-04-20T02:52:45.398 INFO:tasks.rados.rados.0.smithi105.stdout:1720: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:45.398 INFO:tasks.rados.rados.0.smithi105.stdout:1721: set_chunk oid 33 target oid 2024-04-20T02:52:45.420 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:52:45.420+0000 7f6a6fd26700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 43317) UID: 0 2024-04-20T02:52:45.455 INFO:tasks.rados.rados.0.smithi105.stdout:1721: set_chunk oid 33 offset: 458240 length: 793088 target oid 8f8ab78cc85da16d322a687edcfbb2532a6711443ca7f5bf3784b3eecfbdb4ec offset: 0 snap -1 2024-04-20T02:52:45.455 INFO:tasks.rados.rados.0.smithi105.stdout:1722: rollback oid 9 current snap is 36 2024-04-20T02:52:45.456 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 9 to 35 2024-04-20T02:52:45.456 INFO:tasks.rados.rados.0.smithi105.stdout:1706: finishing write tid 3 to smithi10548823-46 2024-04-20T02:52:45.456 INFO:tasks.rados.rados.0.smithi105.stdout:1723: set_chunk oid 28 target oid 2024-04-20T02:52:45.504 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:45.504 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:45.514 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1723: set_chunk oid 28 offset: 516608 length: 2614784 target oid 5b5f18ddd3065d853e550df30e491a050781f2354e482df454666c2733b8e628 offset: 0 snap 35 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1724: write oid 17 current snap is 36 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1707: finishing write tid 1 to smithi10548823-13 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1707: finishing write tid 2 to smithi10548823-13 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1707: finishing write tid 3 to smithi10548823-13 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1707: finishing write tid 4 to smithi10548823-13 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1707: finishing write tid 5 to smithi10548823-13 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1707: oid 13 updating version 0 to 389 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1707: oid 13 updating version 389 to 390 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1707: oid 13 updating version 390 to 392 2024-04-20T02:52:45.529 INFO:tasks.rados.rados.0.smithi105.stdout:1707: oid 13 version 392 is already newer than 391 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 392 (ObjNum 335 snap 36 seq_num 335) dirty exists 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1707: left oid 13 (ObjNum 335 snap 36 seq_num 335) 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1708: finishing write tid 1 to smithi10548823-8 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1706: finishing write tid 4 to smithi10548823-46 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1706: finishing write tid 5 to smithi10548823-46 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1706: oid 46 updating version 0 to 102 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1706: oid 46 updating version 102 to 104 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1706: oid 46 updating version 104 to 105 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1706: oid 46 version 105 is already newer than 103 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 105 (ObjNum 334 snap 36 seq_num 334) dirty exists 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1706: left oid 46 (ObjNum 334 snap 36 seq_num 334) 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1708: finishing write tid 2 to smithi10548823-8 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1708: finishing write tid 3 to smithi10548823-8 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1708: finishing write tid 4 to smithi10548823-8 2024-04-20T02:52:45.530 INFO:tasks.rados.rados.0.smithi105.stdout:1708: finishing write tid 5 to smithi10548823-8 2024-04-20T02:52:45.531 INFO:tasks.rados.rados.0.smithi105.stdout:1708: oid 8 updating version 0 to 135 2024-04-20T02:52:45.531 INFO:tasks.rados.rados.0.smithi105.stdout:1708: oid 8 updating version 135 to 136 2024-04-20T02:52:45.531 INFO:tasks.rados.rados.0.smithi105.stdout:1708: oid 8 version 136 is already newer than 133 2024-04-20T02:52:45.531 INFO:tasks.rados.rados.0.smithi105.stdout:1708: oid 8 version 136 is already newer than 134 2024-04-20T02:52:45.531 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 136 (ObjNum 336 snap 36 seq_num 336) dirty exists 2024-04-20T02:52:45.531 INFO:tasks.rados.rados.0.smithi105.stdout:1708: left oid 8 (ObjNum 336 snap 36 seq_num 336) 2024-04-20T02:52:45.531 INFO:tasks.rados.rados.0.smithi105.stdout:1724: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-04-20T02:52:45.531 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:45.528+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:45.536 INFO:tasks.rados.rados.0.smithi105.stdout:1724: writing smithi10548823-17 from 514380 to 1218572 tid 1 2024-04-20T02:52:45.541 INFO:tasks.rados.rados.0.smithi105.stdout:1724: writing smithi10548823-17 from 1629760 to 2108195 tid 2 2024-04-20T02:52:45.546 INFO:tasks.rados.rados.0.smithi105.stdout:1724: writing smithi10548823-17 from 2604288 to 3046890 tid 3 2024-04-20T02:52:45.546 INFO:tasks.rados.rados.0.smithi105.stdout:1706: done (15 left) 2024-04-20T02:52:45.547 INFO:tasks.rados.rados.0.smithi105.stdout:1707: done (14 left) 2024-04-20T02:52:45.547 INFO:tasks.rados.rados.0.smithi105.stdout:1708: done (13 left) 2024-04-20T02:52:45.547 INFO:tasks.rados.rados.0.smithi105.stdout:1725: write oid 49 current snap is 36 2024-04-20T02:52:45.547 INFO:tasks.rados.rados.0.smithi105.stdout:1725: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-04-20T02:52:45.553 INFO:tasks.rados.rados.0.smithi105.stdout:1725: writing smithi10548823-49 from 531187 to 1310628 tid 1 2024-04-20T02:52:45.562 INFO:tasks.rados.rados.0.smithi105.stdout:1725: writing smithi10548823-49 from 2088242 to 2826688 tid 2 2024-04-20T02:52:45.567 INFO:tasks.rados.rados.0.smithi105.stdout:1725: writing smithi10548823-49 from 3431711 to 3731187 tid 3 2024-04-20T02:52:45.567 INFO:tasks.rados.rados.0.smithi105.stdout:1726: snap_create 2024-04-20T02:52:45.567 INFO:tasks.rados.rados.0.smithi105.stdout:1718: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:45.621 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:45.621+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:45.648 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:45.648 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:45.659 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:45.743 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:45.743+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:45.822 INFO:tasks.rados.rados.0.smithi105.stdout:1715: expect (ObjNum 282 snap 27 seq_num 282) 2024-04-20T02:52:45.824 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:45.824+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:45.827 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:45.827 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:45.838 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:45.924 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:45.925+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:45.970 INFO:tasks.rados.rados.0.smithi105.stdout:1720: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:46.008 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:46.008 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:46.019 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:46.025 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:46.026+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1713: finishing write tid 1 to smithi10548823-5 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1713: finishing write tid 2 to smithi10548823-5 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1713: finishing write tid 3 to smithi10548823-5 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1719: finishing set_chunk to oid 10 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1719: got expected ENOENT (src dne) 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1713: finishing write tid 4 to smithi10548823-5 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1713: finishing write tid 5 to smithi10548823-5 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1713: oid 5 updating version 0 to 200 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1713: oid 5 updating version 200 to 202 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1713: oid 5 version 202 is already newer than 201 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:1713: oid 5 updating version 202 to 203 2024-04-20T02:52:46.044 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 203 (ObjNum 339 snap 36 seq_num 339) dirty exists 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1713: left oid 5 (ObjNum 339 snap 36 seq_num 339) 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1722: finishing rollback tid 1 to smithi10548823-9 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 0 (ObjNum 97 snap 0 seq_num 1) dirty dne 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1721: finishing set_chunk to oid 33 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 50 (ObjNum 196 snap 19 seq_num 196) dirty exists 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1723: finishing set_chunk to oid 28 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1724: finishing write tid 1 to smithi10548823-17 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1724: finishing write tid 2 to smithi10548823-17 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1724: finishing write tid 3 to smithi10548823-17 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1724: finishing write tid 4 to smithi10548823-17 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1724: finishing write tid 5 to smithi10548823-17 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1724: oid 17 updating version 0 to 143 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1724: oid 17 version 143 is already newer than 142 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1724: oid 17 version 143 is already newer than 141 2024-04-20T02:52:46.045 INFO:tasks.rados.rados.0.smithi105.stdout:1724: oid 17 version 143 is already newer than 140 2024-04-20T02:52:46.046 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 17 v 143 (ObjNum 340 snap 36 seq_num 340) dirty exists 2024-04-20T02:52:46.046 INFO:tasks.rados.rados.0.smithi105.stdout:1724: left oid 17 (ObjNum 340 snap 36 seq_num 340) 2024-04-20T02:52:46.126 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:46.126+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:46.138 INFO:tasks.rados.rados.0.smithi105.stdout:1713: done (14 left) 2024-04-20T02:52:46.138 INFO:tasks.rados.rados.0.smithi105.stdout:1715: done (13 left) 2024-04-20T02:52:46.138 INFO:tasks.rados.rados.0.smithi105.stdout:1716: done (12 left) 2024-04-20T02:52:46.139 INFO:tasks.rados.rados.0.smithi105.stdout:1718: done (11 left) 2024-04-20T02:52:46.139 INFO:tasks.rados.rados.0.smithi105.stdout:1719: done (10 left) 2024-04-20T02:52:46.139 INFO:tasks.rados.rados.0.smithi105.stdout:1720: done (9 left) 2024-04-20T02:52:46.139 INFO:tasks.rados.rados.0.smithi105.stdout:1721: done (8 left) 2024-04-20T02:52:46.139 INFO:tasks.rados.rados.0.smithi105.stdout:1722: done (7 left) 2024-04-20T02:52:46.139 INFO:tasks.rados.rados.0.smithi105.stdout:1723: done (6 left) 2024-04-20T02:52:46.139 INFO:tasks.rados.rados.0.smithi105.stdout:1724: done (5 left) 2024-04-20T02:52:46.139 INFO:tasks.rados.rados.0.smithi105.stdout:1726: done (4 left) 2024-04-20T02:52:46.139 INFO:tasks.rados.rados.0.smithi105.stdout:1727: write oid 32 current snap is 37 2024-04-20T02:52:46.139 INFO:tasks.rados.rados.0.smithi105.stdout:1727: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-04-20T02:52:46.141 INFO:tasks.rados.rados.0.smithi105.stdout:1727: writing smithi10548823-32 from 547994 to 1002684 tid 1 2024-04-20T02:52:46.148 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:52:46.149 INFO:tasks.rados.rados.0.smithi105.stdout:1727: writing smithi10548823-32 from 1746724 to 2228828 tid 2 2024-04-20T02:52:46.153 INFO:tasks.rados.rados.0.smithi105.stdout:1727: writing smithi10548823-32 from 2659134 to 3124280 tid 3 2024-04-20T02:52:46.161 INFO:tasks.rados.rados.0.smithi105.stdout:1727: writing smithi10548823-32 from 3605976 to 3747994 tid 4 2024-04-20T02:52:46.161 INFO:tasks.rados.rados.0.smithi105.stdout:1728: write oid 14 current snap is 37 2024-04-20T02:52:46.162 INFO:tasks.rados.rados.0.smithi105.stdout:1728: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-04-20T02:52:46.164 INFO:tasks.rados.rados.0.smithi105.stdout:1728: writing smithi10548823-14 from 564801 to 1211093 tid 1 2024-04-20T02:52:46.172 INFO:tasks.rados.rados.0.smithi105.stdout:1728: writing smithi10548823-14 from 1921559 to 2663674 tid 2 2024-04-20T02:52:46.179 INFO:tasks.rados.rados.0.smithi105.stdout:1728: writing smithi10548823-14 from 3202910 to 3764801 tid 3 2024-04-20T02:52:46.179 INFO:tasks.rados.rados.0.smithi105.stdout:1729: read oid 45 snap -1 2024-04-20T02:52:46.179 INFO:tasks.rados.rados.0.smithi105.stdout:1729: expect (ObjNum 325 snap 35 seq_num 325) 2024-04-20T02:52:46.179 INFO:tasks.rados.rados.0.smithi105.stdout:1730: read oid 5 snap -1 2024-04-20T02:52:46.179 INFO:tasks.rados.rados.0.smithi105.stdout:1730: expect (ObjNum 339 snap 36 seq_num 339) 2024-04-20T02:52:46.179 INFO:tasks.rados.rados.0.smithi105.stdout:1731: write oid 40 current snap is 37 2024-04-20T02:52:46.179 INFO:tasks.rados.rados.0.smithi105.stdout:1731: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-04-20T02:52:46.186 INFO:tasks.rados.rados.0.smithi105.stdout:1731: writing smithi10548823-40 from 581608 to 1303149 tid 1 2024-04-20T02:52:46.193 INFO:tasks.rados.rados.0.smithi105.stdout:1731: writing smithi10548823-40 from 1863688 to 2349461 tid 2 2024-04-20T02:52:46.201 INFO:tasks.rados.rados.0.smithi105.stdout:1731: writing smithi10548823-40 from 3113980 to 3718023 tid 3 2024-04-20T02:52:46.201 INFO:tasks.rados.rados.0.smithi105.stdout:1732: snap_remove snap 33 2024-04-20T02:52:46.209 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:46.209 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:46.227 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:46.228 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:46.227+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:46.327 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:46.328+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:46.400 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:46.400 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:46.411 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:46.429 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:46.429+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:46.529 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:46.529+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:46.539 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:46.539 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:46.550 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:46.634 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:46.634+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:46.719 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:46.719 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:46.729 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:46.771 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:46.771+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:46.832 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:46.832+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:46.898 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:46.898 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:46.908 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:46.949 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:46.949+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:47.034 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:47.035+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:47.076 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:47.076 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:47.086 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:47.135 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:47.135+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:47.136 INFO:tasks.rados.rados.0.smithi105.stdout:1732: done (9 left) 2024-04-20T02:52:47.136 INFO:tasks.rados.rados.0.smithi105.stdout:1733: write oid 13 current snap is 37 2024-04-20T02:52:47.136 INFO:tasks.rados.rados.0.smithi105.stdout:1714: got (22) Invalid argument 2024-04-20T02:52:47.136 INFO:tasks.rados.rados.0.smithi105.stdout:1733: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-04-20T02:52:47.144 INFO:tasks.rados.rados.0.smithi105.stdout:1733: writing smithi10548823-13 from 598415 to 1395205 tid 1 2024-04-20T02:52:47.151 INFO:tasks.rados.rados.0.smithi105.stdout:1733: writing smithi10548823-13 from 1922170 to 2667954 tid 2 2024-04-20T02:52:47.157 INFO:tasks.rados.rados.0.smithi105.stdout:1733: writing smithi10548823-13 from 3141403 to 3556718 tid 3 2024-04-20T02:52:47.157 INFO:tasks.rados.rados.0.smithi105.stdout:1714: done (9 left) 2024-04-20T02:52:47.157 INFO:tasks.rados.rados.0.smithi105.stdout:1734: write (excl) oid 44 current snap is 37 2024-04-20T02:52:47.157 INFO:tasks.rados.rados.0.smithi105.stdout:1734: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-04-20T02:52:47.164 INFO:tasks.rados.rados.0.smithi105.stdout:1734: writing smithi10548823-44 from 615222 to 1087261 tid 1 2024-04-20T02:52:47.170 INFO:tasks.rados.rados.0.smithi105.stdout:1734: writing smithi10548823-44 from 1580652 to 2070094 tid 2 2024-04-20T02:52:47.178 INFO:tasks.rados.rados.0.smithi105.stdout:1734: writing smithi10548823-44 from 2768826 to 3395413 tid 3 2024-04-20T02:52:47.178 INFO:tasks.rados.rados.0.smithi105.stdout:1735: read oid 4 snap -1 2024-04-20T02:52:47.178 INFO:tasks.rados.rados.0.smithi105.stdout:1735: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:47.178 INFO:tasks.rados.rados.0.smithi105.stdout:1736: write oid 11 current snap is 37 2024-04-20T02:52:47.178 INFO:tasks.rados.rados.0.smithi105.stdout:1736: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-04-20T02:52:47.185 INFO:tasks.rados.rados.0.smithi105.stdout:1736: writing smithi10548823-11 from 632029 to 1179317 tid 1 2024-04-20T02:52:47.193 INFO:tasks.rados.rados.0.smithi105.stdout:1736: writing smithi10548823-11 from 1639134 to 2388587 tid 2 2024-04-20T02:52:47.199 INFO:tasks.rados.rados.0.smithi105.stdout:1736: writing smithi10548823-11 from 2796249 to 3234108 tid 3 2024-04-20T02:52:47.202 INFO:tasks.rados.rados.0.smithi105.stdout:1736: writing smithi10548823-11 from 3653524 to 3832029 tid 4 2024-04-20T02:52:47.203 INFO:tasks.rados.rados.0.smithi105.stdout:1737: set_chunk oid 39 target oid 2024-04-20T02:52:47.230 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:47.236 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:47.236+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:47.246 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:47.246 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:47.250 INFO:tasks.rados.rados.0.smithi105.stdout:1737: set_chunk oid 39 offset: 2439680 length: 75264 target oid 2fb38ab15b20400869ecff984e99fb1e5c82b4b555a7f71cbeb4abd2abc8b9f6 offset: 0 snap -1 2024-04-20T02:52:47.250 INFO:tasks.rados.rados.0.smithi105.stdout:1738: read oid 33 snap -1 2024-04-20T02:52:47.251 INFO:tasks.rados.rados.0.smithi105.stdout:1738: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:52:47.251 INFO:tasks.rados.rados.0.smithi105.stdout:1739: tier_promote oid 35 2024-04-20T02:52:47.251 INFO:tasks.rados.rados.0.smithi105.stdout:1740: set_chunk oid 16 target oid 2024-04-20T02:52:47.251 INFO:tasks.rados.rados.0.smithi105.stdout:1740: set_chunk oid 16 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:52:47.251 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:47.253 INFO:tasks.rados.rados.0.smithi105.stdout:1712: expect (ObjNum 305 snap 33 seq_num 305) 2024-04-20T02:52:47.264 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:47.348 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:47.348+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:47.390 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:47.411 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:47.430 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:47.437 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:47.437+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:47.440 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:47.440 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:47.449 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:47.451 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:47.469 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:52:47.492 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:47.495 INFO:tasks.rados.rados.0.smithi105.stdout:1717: expect (ObjNum 307 snap 33 seq_num 307) 2024-04-20T02:52:47.511 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:47.531 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:47.532 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:47.532 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:47.532 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:47.532 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:47.532 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:52:47.532 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:47.532 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:47.532 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:47.538 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:47.538+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:47.603 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 121.057885449, 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.604 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:52:47.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:52:47.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:969 4.0 4:007b2a87:::smithi10548823-39:head [write 1681102~474010] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 121.051664375, 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057530313, 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.610 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 969 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549961+0000", 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.549957+0000", 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999967 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550087+0000", 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000130004 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550107+0000", 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9454999999999999e-05 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.550117+0000", 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7380000000000006e-06 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564679+0000", 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01456207 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.564817+0000", 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013822699999999999 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.565027+0000", 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020990399999999999 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594624+0000", 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029596764000000001 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595318+0000", 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00069462500000000004 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607418+0000", 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012099611999999999 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607456+0000", 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8216000000000001e-05 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607491+0000", 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.4885000000000002e-05 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1209 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 463893~562240,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 113.236599523, 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058429782999999999, 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.616 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1209 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365025+0000", 2024-04-20T02:52:47.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365023+0000", 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365327+0000", 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030412100000000002 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365350+0000", 2024-04-20T02:52:47.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2823000000000002e-05 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365377+0000", 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7084999999999999e-05 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365445+0000", 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7553999999999997e-05 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365521+0000", 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6636000000000003e-05 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.365675+0000", 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000153549 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.406973+0000", 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041298031999999998 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423306+0000", 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016333295000000001 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423347+0000", 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0806000000000002e-05 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423404+0000", 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7499e-05 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423455+0000", 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0840000000000001e-05 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 113.229336561, 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.622 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:52:47.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:52:47.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1211 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 2685935~559341] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 113.223394032, 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057141839999999999, 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1211 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378231+0000", 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378229+0000", 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378391+0000", 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016169900000000001 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378408+0000", 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6827999999999999e-05 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378425+0000", 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.696e-05 2024-04-20T02:52:47.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407020+0000", 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.028595033999999998 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407138+0000", 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011820900000000001 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.407299+0000", 2024-04-20T02:52:47.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000161154 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428395+0000", 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021096539000000001 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.433711+0000", 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053153469999999998 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435262+0000", 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001551296 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435322+0000", 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9483e-05 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435373+0000", 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1171000000000002e-05 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1212 4.0 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (56) in=7b,truncate 3663893] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 113.22319104, 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057081708000000002, 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1212 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378434+0000", 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378433+0000", 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378438+0000", 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7720000000000002e-06 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378446+0000", 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6520000000000006e-06 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378452+0000", 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0959999999999997e-06 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423494+0000", 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045042470000000001 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423608+0000", 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114017 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423791+0000", 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018271100000000001 2024-04-20T02:52:47.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.428403+0000", 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0046124390000000003 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435229+0000", 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.006825472 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435413+0000", 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00018405799999999999 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435458+0000", 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5048e-05 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435516+0000", 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7639000000000002e-05 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.640 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1213 4.0 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 113.223168113, 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057295458, 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1213 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.641 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378457+0000", 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378456+0000", 2024-04-20T02:52:47.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378459+0000", 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8449999999999999e-06 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378467+0000", 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9400000000000002e-06 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.378474+0000", 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5989999999999998e-06 2024-04-20T02:52:47.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424091+0000", 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.045616761999999998 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424191+0000", 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000100855 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435547+0000", 2024-04-20T02:52:47.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011355522999999999 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435649+0000", 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010211 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.435752+0000", 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103281 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 113.21308437899999, 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:52:47.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1659 3.2 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head [write 0~2962944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 98.155367057999996, 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.055317244000000002, 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.650 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1659 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446258+0000", 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.446255+0000", 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447670+0000", 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014145029999999999 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447691+0000", 2024-04-20T02:52:47.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.1599000000000001e-05 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.447702+0000", 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1005000000000001e-05 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448177+0000", 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00047437899999999998 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448320+0000", 2024-04-20T02:52:47.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000143239 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.448427+0000", 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010665700000000001 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.496041+0000", 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047614744000000001 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501432+0000", 2024-04-20T02:52:47.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0053906090000000002 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501519+0000", 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7349999999999995e-05 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501552+0000", 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2515999999999998e-05 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:09.501575+0000", 2024-04-20T02:52:47.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3360999999999999e-05 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2045 4.11 4:8b927dd6:::smithi10548823-28:head [write 642067~696188] snapc 19=[18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 82.841051281999995, 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.056553434999999999, 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.656 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2045 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760574+0000", 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760571+0000", 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760935+0000", 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036386200000000001 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760962+0000", 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7149e-05 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.760977+0000", 2024-04-20T02:52:47.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4856e-05 2024-04-20T02:52:47.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761052+0000", 2024-04-20T02:52:47.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5748999999999996e-05 2024-04-20T02:52:47.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761251+0000", 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019845999999999999 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.761509+0000", 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000258249 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804571+0000", 2024-04-20T02:52:47.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043061599999999998 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.804612+0000", 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1e-05 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817023+0000", 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012411159999999999 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817071+0000", 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7769999999999998e-05 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:24.817127+0000", 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6560000000000001e-05 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.662 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.253385360000003, 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:47.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2700 4.0 4:007b2a87:::smithi10548823-39:head [write 1967200~714747] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.244895165999999, 2024-04-20T02:52:47.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061846287, 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2700 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356730+0000", 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356728+0000", 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356879+0000", 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000150958 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356894+0000", 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5132e-05 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.356906+0000", 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2071e-05 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366666+0000", 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0097595289999999994 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366774+0000", 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108527 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.366956+0000", 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000181359 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390515+0000", 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023559902000000001 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406790+0000", 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016274555999999999 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418488+0000", 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011697591 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418529+0000", 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1545000000000003e-05 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418576+0000", 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6903000000000001e-05 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2701 4.0 4:007b2a87:::smithi10548823-39:head [write 3421983~426820] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.237719173999999, 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054799176999999998, 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.674 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2701 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363906+0000", 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.363904+0000", 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364154+0000", 2024-04-20T02:52:47.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000250179 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364171+0000", 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6748e-05 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364182+0000", 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1371e-05 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390552+0000", 2024-04-20T02:52:47.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026369607 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390652+0000", 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010059499999999999 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390857+0000", 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020445199999999999 2024-04-20T02:52:47.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418224+0000", 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027367316999999999 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418282+0000", 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8279000000000002e-05 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418618+0000", 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000335967 2024-04-20T02:52:47.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418661+0000", 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2614999999999998e-05 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418705+0000", 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3993999999999999e-05 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2702 4.0 4:007b2a87:::smithi10548823-39:head [setxattr _header (57) in=7b,truncate 3848803] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.237433119999999, 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058669165000000002, 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2702 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364192+0000", 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364191+0000", 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364196+0000", 2024-04-20T02:52:47.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4490000000000001e-06 2024-04-20T02:52:47.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364201+0000", 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5040000000000002e-06 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364206+0000", 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.405895+0000", 2024-04-20T02:52:47.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041689047 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406021+0000", 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000125561 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406541+0000", 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052000699999999996 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418451+0000", 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011910041999999999 2024-04-20T02:52:47.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418747+0000", 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00029622399999999999 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422704+0000", 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0039571859999999997 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422781+0000", 2024-04-20T02:52:47.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.661e-05 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422861+0000", 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9987999999999994e-05 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2703 4.0 4:007b2a87:::smithi10548823-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.237414068, 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.058940278999999998, 2024-04-20T02:52:47.687 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2703 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364211+0000", 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364213+0000", 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4830000000000002e-06 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364220+0000", 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.007e-06 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.364225+0000", 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0350000000000001e-06 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406820+0000", 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.042595135999999999 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.406923+0000", 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000102846 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.422897+0000", 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.015974202999999999 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423015+0000", 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117581 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.423151+0000", 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013635899999999999 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3872 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 779924~785847,stat] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.2509821209999998, 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070878165000000007, 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.692 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3872 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350641+0000", 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351179+0000", 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053797199999999995 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351196+0000", 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.66e-05 2024-04-20T02:52:47.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351209+0000", 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2952000000000001e-05 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351306+0000", 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7272e-05 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351322+0000", 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6416e-05 2024-04-20T02:52:47.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358164+0000", 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068416019999999996 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358336+0000", 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017156499999999999 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358594+0000", 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025794499999999998 2024-04-20T02:52:47.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389148+0000", 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030554773 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409688+0000", 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020539497 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420893+0000", 2024-04-20T02:52:47.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011205503 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420934+0000", 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1171000000000003e-05 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420998+0000", 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3386999999999995e-05 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421521+0000", 2024-04-20T02:52:47.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052313200000000005 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3873 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 2078257~580112] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.2503988349999999, 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075737411000000004, 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.699 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3873 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351225+0000", 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351614+0000", 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038825099999999999 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351625+0000", 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1173e-05 2024-04-20T02:52:47.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351634+0000", 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0489999999999993e-06 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351696+0000", 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1885999999999997e-05 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:47.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351708+0000", 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2702e-05 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389193+0000", 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037484574999999999 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389303+0000", 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110188 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:47.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389563+0000", 2024-04-20T02:52:47.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026034499999999999 2024-04-20T02:52:47.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420846+0000", 2024-04-20T02:52:47.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031283014999999997 2024-04-20T02:52:47.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421592+0000", 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074529299999999995 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421632+0000", 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0677999999999999e-05 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426683+0000", 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050507119999999997 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426752+0000", 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8484000000000004e-05 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426964+0000", 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021193500000000001 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3874 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 3301257~278667] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.2499638339999999, 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075720365999999997, 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3874 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351660+0000", 2024-04-20T02:52:47.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351852+0000", 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019116900000000001 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351861+0000", 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6050000000000004e-06 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351869+0000", 2024-04-20T02:52:47.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4780000000000002e-06 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351916+0000", 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7222999999999998e-05 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351927+0000", 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0791000000000001e-05 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409721+0000", 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057793887000000002 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409840+0000", 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119837 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.410063+0000", 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022257200000000001 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421678+0000", 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011615448 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421721+0000", 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2191000000000003e-05 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.423989+0000", 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022682879999999998 2024-04-20T02:52:47.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427051+0000", 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003061785 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427157+0000", 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106684 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427381+0000", 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022416299999999999 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3875 4.10 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (57) in=7b,truncate 3579924] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.2497426730000001, 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075851731000000006, 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:47.714 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3875 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351886+0000", 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.561e-06 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351893+0000", 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6100000000000002e-06 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351899+0000", 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8660000000000002e-06 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352002+0000", 2024-04-20T02:52:47.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103811 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352012+0000", 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4110000000000002e-06 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421749+0000", 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.069736690000000004 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421861+0000", 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112587 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422120+0000", 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025878099999999997 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424547+0000", 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024270379999999999 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424960+0000", 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041259899999999998 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426257+0000", 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001297231 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427489+0000", 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012320949999999999 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427562+0000", 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2825999999999997e-05 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427734+0000", 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017230699999999999 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3876 4.10 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 3.2497207669999999, 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076230629999999994, 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3876 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:47.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351906+0000", 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6350000000000002e-06 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351912+0000", 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8209999999999998e-06 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351918+0000", 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3839999999999996e-06 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352035+0000", 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117207 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352043+0000", 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0169999999999996e-06 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422511+0000", 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070467815000000003 2024-04-20T02:52:47.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422619+0000", 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010819600000000001 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427772+0000", 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005153398 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427900+0000", 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012740399999999999 2024-04-20T02:52:47.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:47.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:47.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:47.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.428135+0000", 2024-04-20T02:52:47.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235138 2024-04-20T02:52:47.727 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.727 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.727 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.727 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:47.727 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:47.727 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:47.729 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:52:47.729 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:47.729+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:47.769 INFO:tasks.rados.rados.0.smithi105.stdout:1727: finishing write tid 1 to smithi10548823-32 2024-04-20T02:52:47.769 INFO:tasks.rados.rados.0.smithi105.stdout:1727: finishing write tid 2 to smithi10548823-32 2024-04-20T02:52:47.769 INFO:tasks.rados.rados.0.smithi105.stdout:1730: expect (ObjNum 339 snap 36 seq_num 339) 2024-04-20T02:52:47.829 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:47.829+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:47.859 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:47.860 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:47.860 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:47.860 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:47.870 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:52:47.930 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:47.930+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:47.952 INFO:tasks.rados.rados.0.smithi105.stdout:1727: finishing write tid 3 to smithi10548823-32 2024-04-20T02:52:47.952 INFO:tasks.rados.rados.0.smithi105.stdout:1727: finishing write tid 4 to smithi10548823-32 2024-04-20T02:52:47.952 INFO:tasks.rados.rados.0.smithi105.stdout:1727: finishing write tid 5 to smithi10548823-32 2024-04-20T02:52:47.952 INFO:tasks.rados.rados.0.smithi105.stdout:1727: finishing write tid 6 to smithi10548823-32 2024-04-20T02:52:47.952 INFO:tasks.rados.rados.0.smithi105.stdout:1727: oid 32 updating version 0 to 92 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1727: oid 32 version 92 is already newer than 91 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1727: oid 32 version 92 is already newer than 89 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1727: oid 32 updating version 92 to 93 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1727: oid 32 version 93 is already newer than 90 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 93 (ObjNum 342 snap 37 seq_num 342) dirty exists 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1727: left oid 32 (ObjNum 342 snap 37 seq_num 342) 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1728: finishing write tid 1 to smithi10548823-14 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1712: done (15 left) 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1717: done (14 left) 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1727: done (13 left) 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1730: done (12 left) 2024-04-20T02:52:47.953 INFO:tasks.rados.rados.0.smithi105.stdout:1741: set_chunk oid 26 target oid 2024-04-20T02:52:47.954 INFO:tasks.rados.rados.0.smithi105.stdout:1741: set_chunk oid 26 offset: 480768 length: 25088 target oid d82ad1081dfe2e318b6d888666acf1f5edde7d6dfbfebfc53ffaf52dcff90a0b offset: 0 snap -1 2024-04-20T02:52:47.954 INFO:tasks.rados.rados.0.smithi105.stdout:1742: write oid 36 current snap is 37 2024-04-20T02:52:47.954 INFO:tasks.rados.rados.0.smithi105.stdout:1725: finishing write tid 1 to smithi10548823-49 2024-04-20T02:52:47.954 INFO:tasks.rados.rados.0.smithi105.stdout:1742: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-04-20T02:52:47.961 INFO:tasks.rados.rados.0.smithi105.stdout:1742: writing smithi10548823-36 from 648836 to 1271373 tid 1 2024-04-20T02:52:47.968 INFO:tasks.rados.rados.0.smithi105.stdout:1742: writing smithi10548823-36 from 1981263 to 2474374 tid 2 2024-04-20T02:52:47.978 INFO:tasks.rados.rados.0.smithi105.stdout:1742: writing smithi10548823-36 from 3107319 to 3848836 tid 3 2024-04-20T02:52:47.978 INFO:tasks.rados.rados.0.smithi105.stdout:1743: read oid 15 snap -1 2024-04-20T02:52:47.978 INFO:tasks.rados.rados.0.smithi105.stdout:1743: expect deleted 2024-04-20T02:52:47.978 INFO:tasks.rados.rados.0.smithi105.stdout:1744: set_chunk oid 30 target oid 2024-04-20T02:52:48.014 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:48.014 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:48.014 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:48.014 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:48.014 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:48.021 INFO:tasks.rados.rados.0.smithi105.stdout:1744: set_chunk oid 30 offset: 1056768 length: 1410560 target oid 6c2cde9ff0b0daf2b5b576bba4cf63f818eef78cae00cf84a226b251de37e0e1 offset: 0 snap -1 2024-04-20T02:52:48.021 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:48.021 INFO:tasks.rados.rados.0.smithi105.stdout:1725: finishing write tid 2 to smithi10548823-49 2024-04-20T02:52:48.021 INFO:tasks.rados.rados.0.smithi105.stdout:1725: finishing write tid 3 to smithi10548823-49 2024-04-20T02:52:48.021 INFO:tasks.rados.rados.0.smithi105.stdout:1725: finishing write tid 4 to smithi10548823-49 2024-04-20T02:52:48.022 INFO:tasks.rados.rados.0.smithi105.stdout:1725: finishing write tid 5 to smithi10548823-49 2024-04-20T02:52:48.022 INFO:tasks.rados.rados.0.smithi105.stdout:1725: oid 49 updating version 0 to 231 2024-04-20T02:52:48.022 INFO:tasks.rados.rados.0.smithi105.stdout:1725: oid 49 version 231 is already newer than 229 2024-04-20T02:52:48.022 INFO:tasks.rados.rados.0.smithi105.stdout:1725: oid 49 version 231 is already newer than 230 2024-04-20T02:52:48.022 INFO:tasks.rados.rados.0.smithi105.stdout:1725: oid 49 updating version 231 to 236 2024-04-20T02:52:48.022 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 236 (ObjNum 341 snap 36 seq_num 341) dirty exists 2024-04-20T02:52:48.022 INFO:tasks.rados.rados.0.smithi105.stdout:1725: left oid 49 (ObjNum 341 snap 36 seq_num 341) 2024-04-20T02:52:48.022 INFO:tasks.rados.rados.0.smithi105.stdout:1725: done (15 left) 2024-04-20T02:52:48.022 INFO:tasks.rados.rados.0.smithi105.stdout:1745: set_chunk oid 23 target oid 2024-04-20T02:52:48.027 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:52:48.031 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:48.031+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:48.050 INFO:tasks.rados.rados.0.smithi105.stdout:1745: set_chunk oid 23 offset: 1542144 length: 998400 target oid ac59d86a6fa6edc585095e0c7089d8b4a7b553c865deda6c1b09edaf9d902c3b offset: 0 snap -1 2024-04-20T02:52:48.050 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:48.050 INFO:tasks.rados.rados.0.smithi105.stdout:1729: expect (ObjNum 325 snap 35 seq_num 325) 2024-04-20T02:52:48.132 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:48.132+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:48.157 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1144 4.d 4:b376c20e:::smithi10548823-49:head [write 763051~794393,stat] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.21373599899999, 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082570415999999994, 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.158 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1144 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942977+0000", 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.942975+0000", 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944610+0000", 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0016346119999999999 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944628+0000", 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8567999999999999e-05 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944644+0000", 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5688999999999998e-05 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944709+0000", 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5458999999999998e-05 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944816+0000", 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106543 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.944998+0000", 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018190400000000001 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983895+0000", 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038897462000000001 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.986802+0000", 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029069120000000002 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025467+0000", 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038664506000000001 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025508+0000", 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1754e-05 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025547+0000", 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8937999999999999e-05 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1145 4.d 4:b376c20e:::smithi10548823-49:head [write 2158943~647020] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.206257681, 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075196244999999995, 2024-04-20T02:52:48.164 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1145 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950455+0000", 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.950453+0000", 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951166+0000", 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00071236100000000001 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951182+0000", 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5872e-05 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.951193+0000", 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1464000000000001e-05 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963561+0000", 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012368084999999999 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963694+0000", 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013271600000000001 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,2", 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.963861+0000", 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000167539 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.984307+0000", 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020446015000000001 2024-04-20T02:52:48.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.990948+0000", 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0066402220000000003 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025591+0000", 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034642851000000002 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025625+0000", 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4594000000000001e-05 2024-04-20T02:52:48.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025652+0000", 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6438999999999999e-05 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1148 4.d 4:b376c20e:::smithi10548823-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 117.201499782, 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070868167999999995, 2024-04-20T02:52:48.170 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1148 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955213+0000", 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955215+0000", 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0320000000000002e-06 2024-04-20T02:52:48.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955218+0000", 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0970000000000002e-06 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.955221+0000", 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0299999999999998e-06 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983911+0000", 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.028690340000000002 2024-04-20T02:52:48.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:50.983980+0000", 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.8332999999999998e-05 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.025939+0000", 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041959620000000003 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026016+0000", 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6511e-05 2024-04-20T02:52:48.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:51.026081+0000", 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5455000000000003e-05 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1218 4.1 4:8b927dd6:::smithi10548823-28:head [rollback 6] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.778070848, 2024-04-20T02:52:48.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090951765000000004, 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1218 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378642+0000", 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378645+0000", 2024-04-20T02:52:48.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.818e-06 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378651+0000", 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.754e-06 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378656+0000", 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0880000000000002e-06 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378755+0000", 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9025999999999997e-05 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378763+0000", 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.824e-06 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389337+0000", 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010573808000000001 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389441+0000", 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000103835 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.389723+0000", 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00028197200000000002 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405094+0000", 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.015371668 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405296+0000", 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020126700000000001 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469482+0000", 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064185848000000004 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469537+0000", 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5906000000000003e-05 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469594+0000", 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6428999999999998e-05 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1219 4.1 4:8b927dd6:::smithi10548823-28:head [append 0~0] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.77803396500001, 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.091009044999999997, 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.182 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1219 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378679+0000", 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378682+0000", 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1159999999999999e-06 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378688+0000", 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7100000000000001e-06 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378698+0000", 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3729999999999999e-06 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378799+0000", 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100969 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378802+0000", 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2509999999999999e-06 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390803+0000", 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012000673999999999 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.390917+0000", 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000114627 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391060+0000", 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000142192 2024-04-20T02:52:48.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405126+0000", 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.014066786 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405351+0000", 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00022450599999999999 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469601+0000", 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064250606000000002 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469647+0000", 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469688+0000", 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0788999999999999e-05 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.188 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1220 4.9 4:964390c0:::smithi10548823-36:head [tier-promote] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.778009332, 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.110460323, 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1220 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378704+0000", 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378703+0000", 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378706+0000", 2024-04-20T02:52:48.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6019999999999998e-06 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378712+0000", 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.533e-06 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378717+0000", 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7759999999999997e-06 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378795+0000", 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8149e-05 2024-04-20T02:52:48.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op must wait for map", 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.378801+0000", 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9510000000000003e-06 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387623+0000", 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0088213530000000005 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387708+0000", 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4969000000000006e-05 2024-04-20T02:52:48.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489164+0000", 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.101456437 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1221 4.9 4:95e19a88:::smithi10548823-40:head [write 480700~637489,stat] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.771515921, 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084824969, 2024-04-20T02:52:48.193 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1221 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385197+0000", 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385196+0000", 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385856+0000", 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065992600000000002 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385868+0000", 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1673e-05 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385875+0000", 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7959999999999998e-06 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385915+0000", 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0039000000000001e-05 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting_for_map not empty", 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.385919+0000", 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7670000000000003e-06 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387867+0000", 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019474 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.387926+0000", 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8668000000000001e-05 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:48.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.388433+0000", 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050774700000000002 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469906+0000", 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081472463999999994 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469938+0000", 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1939e-05 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469965+0000", 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7759e-05 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469996+0000", 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0647e-05 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470022+0000", 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5982999999999998e-05 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.199 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1222 4.9 4:95e19a88:::smithi10548823-40:head [write 1651964~401685] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.765451899, 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.097772972, 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1222 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391261+0000", 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391259+0000", 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:48.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391641+0000", 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00038180199999999999 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391652+0000", 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.06e-05 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.391661+0000", 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1870000000000003e-06 2024-04-20T02:52:48.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405049+0000", 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013388133999999999 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405112+0000", 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3386000000000006e-05 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.405238+0000", 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012508699999999999 2024-04-20T02:52:48.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470034+0000", 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.064796721000000002 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470069+0000", 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5225999999999998e-05 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.488971+0000", 2024-04-20T02:52:48.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.018901926999999999 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489005+0000", 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3989000000000003e-05 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489034+0000", 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8679000000000001e-05 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1223 4.9 4:95e19a88:::smithi10548823-40:head [write 2713358~770613] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.755390134, 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088130527, 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1223 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401323+0000", 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401322+0000", 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.401991+0000", 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066929000000000001 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402001+0000", 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0386e-05 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402009+0000", 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2300000000000002e-06 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451856+0000", 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049847416999999998 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.451966+0000", 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000109838 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.452157+0000", 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019172500000000001 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472648+0000", 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020490492999999999 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476279+0000", 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00363058 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489400+0000", 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013121539999999999 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489429+0000", 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9243000000000001e-05 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489453+0000", 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.406e-05 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.211 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1224 4.9 4:95e19a88:::smithi10548823-40:head [setxattr _header (56) in=7b,truncate 3680700] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.75469753, 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087516812999999999, 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1224 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:48.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402015+0000", 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402018+0000", 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2490000000000001e-06 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402021+0000", 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1760000000000002e-06 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402024+0000", 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0230000000000001e-06 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.214 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469499+0000", 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067474747000000002 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469593+0000", 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3657000000000001e-05 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.469748+0000", 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015566199999999999 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.472810+0000", 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0030621480000000002 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.476298+0000", 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0034871530000000002 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489476+0000", 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013178779 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489502+0000", 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5947000000000001e-05 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489532+0000", 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0017000000000001e-05 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1225 4.9 4:95e19a88:::smithi10548823-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.754685876, 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.087640739999999995, 2024-04-20T02:52:48.217 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1225 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402027+0000", 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402029+0000", 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.699e-06 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.219 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402032+0000", 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2949999999999998e-06 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.402035+0000", 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7159999999999999e-06 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470088+0000", 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.068053632000000003 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.470139+0000", 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1242999999999997e-05 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489557+0000", 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019418017999999999 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489609+0000", 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1539000000000002e-05 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:54.489668+0000", 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8891000000000002e-05 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1424 4.c 4:392c1e2f:::smithi10548823-44:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 108.04862132, 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080900661999999998, 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.222 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1424 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108092+0000", 2024-04-20T02:52:48.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108090+0000", 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108098+0000", 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9969999999999998e-06 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108111+0000", 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3084e-05 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108138+0000", 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6214999999999999e-05 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108210+0000", 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.2317000000000006e-05 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.108331+0000", 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120621 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188992+0000", 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080661801000000005 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1433 4.1 4:8b91c759:::smithi10548823-5:head [write 450454~642026] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:48.226 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 108.029560604, 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074433145000000006, 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1433 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127152+0000", 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.127151+0000", 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128013+0000", 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086219499999999995 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128028+0000", 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4394e-05 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.128037+0000", 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9449999999999999e-06 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170004+0000", 2024-04-20T02:52:48.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041967222999999998 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170156+0000", 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015219800000000001 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.170356+0000", 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000199581 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188185+0000", 2024-04-20T02:52:48.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017829106000000001 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201483+0000", 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013297995999999999 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201492+0000", 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2699999999999993e-06 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201539+0000", 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201586+0000", 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6099e-05 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1434 4.1 4:8b91c759:::smithi10548823-5:head [write 1505862~544592] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 108.02361429, 2024-04-20T02:52:48.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075830350000000005, 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1434 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133099+0000", 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133097+0000", 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133634+0000", 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053682899999999995 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133646+0000", 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2014000000000001e-05 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133655+0000", 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7730000000000005e-06 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188208+0000", 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.05455339 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188301+0000", 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3029000000000007e-05 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.188501+0000", 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020013999999999999 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201616+0000", 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013114179 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201624+0000", 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5019999999999992e-06 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208856+0000", 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072321360000000001 2024-04-20T02:52:48.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208887+0000", 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0979000000000003e-05 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.208929+0000", 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.1894000000000002e-05 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1435 4.1 4:8b91c759:::smithi10548823-5:head [setxattr _header (55) in=7b,truncate 2050454] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 108.02304884900001, 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077714990999999997, 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1435 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133664+0000", 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133674+0000", 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0601000000000001e-05 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133680+0000", 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8420000000000001e-06 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133685+0000", 2024-04-20T02:52:48.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4890000000000003e-06 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200589+0000", 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066903527000000004 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200689+0000", 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000100467 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.200849+0000", 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015962499999999999 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201627+0000", 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00077819 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201753+0000", 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125406 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211297+0000", 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095440690000000005 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211329+0000", 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2475999999999999e-05 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211379+0000", 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9920000000000003e-05 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.244 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1436 4.1 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 108.023022818, 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077834088999999995, 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1436 2024-04-20T02:52:48.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133690+0000", 2024-04-20T02:52:48.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133692+0000", 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.255e-06 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133697+0000", 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1259999999999997e-06 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.133702+0000", 2024-04-20T02:52:48.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7119999999999999e-06 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201374+0000", 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067672369999999996 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.201437+0000", 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3038000000000002e-05 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211413+0000", 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0099756840000000003 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211462+0000", 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9313e-05 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.211524+0000", 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1995000000000005e-05 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.249 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2837 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.020810017000002, 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99458757099999995, 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.250 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2837 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135903+0000", 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.135897+0000", 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999938 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136552+0000", 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00065543399999999999 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136566+0000", 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4110999999999999e-05 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136578+0000", 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1528999999999999e-05 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136617+0000", 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9051999999999998e-05 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.136625+0000", 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0290000000000005e-06 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130477+0000", 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.993852085 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130491+0000", 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3668e-05 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2838 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:48.254 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.015460492999999, 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98929057499999995, 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2838 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141253+0000", 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141251+0000", 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141685+0000", 2024-04-20T02:52:48.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043407500000000002 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141697+0000", 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1984000000000001e-05 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141708+0000", 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0576e-05 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141755+0000", 2024-04-20T02:52:48.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7024999999999998e-05 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141761+0000", 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7050000000000002e-06 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130533+0000", 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877187099999997 2024-04-20T02:52:48.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130543+0000", 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.8980000000000008e-06 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2839 4.18 4.42a71638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.014990535999999, 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98885721999999998, 2024-04-20T02:52:48.259 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2839 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:48.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141722+0000", 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141726+0000", 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2590000000000002e-06 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141732+0000", 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0889999999999996e-06 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141739+0000", 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9800000000000001e-06 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141790+0000", 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0701999999999997e-05 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141794+0000", 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0999999999999997e-06 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130570+0000", 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98877621599999999 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130580+0000", 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4690000000000003e-06 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.263 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2840 4.18 4.42a71638 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 52.014968000000003, 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98889438799999996, 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2840 2024-04-20T02:52:48.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141745+0000", 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141748+0000", 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.095e-06 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141754+0000", 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1140000000000002e-06 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141759+0000", 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4709999999999998e-06 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141807+0000", 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7487000000000001e-05 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:56.141810+0000", 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8310000000000001e-06 2024-04-20T02:52:48.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130627+0000", 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.98881786999999999 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:57.130639+0000", 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.268 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:48.269 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:52:48.271 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1728: finishing write tid 2 to smithi10548823-14 2024-04-20T02:52:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1728: finishing write tid 3 to smithi10548823-14 2024-04-20T02:52:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1728: finishing write tid 4 to smithi10548823-14 2024-04-20T02:52:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1728: finishing write tid 5 to smithi10548823-14 2024-04-20T02:52:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1728: oid 14 updating version 0 to 125 2024-04-20T02:52:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1728: oid 14 updating version 125 to 127 2024-04-20T02:52:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1728: oid 14 updating version 127 to 128 2024-04-20T02:52:48.272 INFO:tasks.rados.rados.0.smithi105.stdout:1728: oid 14 version 128 is already newer than 126 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 128 (ObjNum 343 snap 37 seq_num 343) dirty exists 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1728: left oid 14 (ObjNum 343 snap 37 seq_num 343) 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1731: finishing write tid 1 to smithi10548823-40 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1731: finishing write tid 2 to smithi10548823-40 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1728: done (15 left) 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1729: done (14 left) 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1746: write oid 47 current snap is 37 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1731: finishing write tid 3 to smithi10548823-40 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1731: finishing write tid 4 to smithi10548823-40 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1731: finishing write tid 5 to smithi10548823-40 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1731: oid 40 updating version 0 to 411 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1731: oid 40 version 411 is already newer than 410 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1731: oid 40 updating version 411 to 412 2024-04-20T02:52:48.273 INFO:tasks.rados.rados.0.smithi105.stdout:1731: oid 40 updating version 412 to 413 2024-04-20T02:52:48.274 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 413 (ObjNum 344 snap 37 seq_num 344) dirty exists 2024-04-20T02:52:48.274 INFO:tasks.rados.rados.0.smithi105.stdout:1731: left oid 40 (ObjNum 344 snap 37 seq_num 344) 2024-04-20T02:52:48.274 INFO:tasks.rados.rados.0.smithi105.stdout:1746: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-04-20T02:52:48.274 INFO:tasks.rados.rados.0.smithi105.stdout:1746: writing smithi10548823-47 from 665643 to 1363429 tid 1 2024-04-20T02:52:48.274 INFO:tasks.rados.rados.0.smithi105.stdout:1746: writing smithi10548823-47 from 2039745 to 2792867 tid 2 2024-04-20T02:52:48.274 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:48.270+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:48.275 INFO:tasks.rados.rados.0.smithi105.stdout:1746: writing smithi10548823-47 from 3534742 to 3865643 tid 3 2024-04-20T02:52:48.275 INFO:tasks.rados.rados.0.smithi105.stdout:1731: done (14 left) 2024-04-20T02:52:48.275 INFO:tasks.rados.rados.0.smithi105.stdout:1747: read oid 22 snap -1 2024-04-20T02:52:48.275 INFO:tasks.rados.rados.0.smithi105.stdout:1747: expect (ObjNum 292 snap 32 seq_num 292) 2024-04-20T02:52:48.275 INFO:tasks.rados.rados.0.smithi105.stdout:1748: write (excl) oid 50 current snap is 37 2024-04-20T02:52:48.275 INFO:tasks.rados.rados.0.smithi105.stdout:1748: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-04-20T02:52:48.279 INFO:tasks.rados.rados.0.smithi105.stdout:1748: writing smithi10548823-50 from 682450 to 1171838 tid 1 2024-04-20T02:52:48.286 INFO:tasks.rados.rados.0.smithi105.stdout:1748: writing smithi10548823-50 from 1814580 to 2311360 tid 2 2024-04-20T02:52:48.293 INFO:tasks.rados.rados.0.smithi105.stdout:1748: writing smithi10548823-50 from 2878518 to 3666546 tid 3 2024-04-20T02:52:48.294 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:48.294 INFO:tasks.rados.rados.0.smithi105.stdout:1735: expect (ObjNum 191 snap 19 seq_num 191) 2024-04-20T02:52:48.371 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:48.371+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:48.417 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:48.418 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:48.418 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:48.418 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:48.428 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:48.472 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:48.472+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:48.481 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 6, 7, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2024-04-20T02:52:48.482 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:52:48.482 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-04-20T02:52:48.482 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 3, 4, 6, 7, 5, 2] 2024-04-20T02:52:48.482 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-20T02:52:48.482 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:52:48.524 INFO:tasks.ceph.osd.4:Stopped 2024-04-20T02:52:48.524 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 4 2024-04-20T02:52:48.549 INFO:tasks.rados.rados.0.smithi105.stdout:1734: finishing write tid 1 to smithi10548823-44 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1733: finishing write tid 1 to smithi10548823-13 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1734: finishing write tid 2 to smithi10548823-44 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1733: finishing write tid 2 to smithi10548823-13 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1733: finishing write tid 3 to smithi10548823-13 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1733: finishing write tid 4 to smithi10548823-13 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1735: done (15 left) 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1749: write (excl) oid 4 current snap is 37 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1733: finishing write tid 5 to smithi10548823-13 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1733: oid 13 updating version 0 to 415 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1733: oid 13 updating version 415 to 421 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1733: oid 13 version 421 is already newer than 420 2024-04-20T02:52:48.550 INFO:tasks.rados.rados.0.smithi105.stdout:1733: oid 13 updating version 421 to 422 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 422 (ObjNum 345 snap 37 seq_num 345) dirty exists 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1733: left oid 13 (ObjNum 345 snap 37 seq_num 345) 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1734: finishing write tid 3 to smithi10548823-44 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1734: finishing write tid 4 to smithi10548823-44 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1734: finishing write tid 5 to smithi10548823-44 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1734: oid 44 updating version 0 to 127 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1734: oid 44 version 127 is already newer than 126 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1734: oid 44 version 127 is already newer than 125 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1734: oid 44 version 127 is already newer than 124 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 127 (ObjNum 346 snap 37 seq_num 346) dirty exists 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1734: left oid 44 (ObjNum 346 snap 37 seq_num 346) 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1736: finishing write tid 1 to smithi10548823-11 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1736: finishing write tid 2 to smithi10548823-11 2024-04-20T02:52:48.551 INFO:tasks.rados.rados.0.smithi105.stdout:1749: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-04-20T02:52:48.554 INFO:tasks.rados.rados.0.smithi105.stdout:1749: writing smithi10548823-4 from 699257 to 1263894 tid 1 2024-04-20T02:52:48.560 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:48.560 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:48.560 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:48.561 INFO:tasks.rados.rados.0.smithi105.stdout:1749: writing smithi10548823-4 from 1873062 to 2513500 tid 2 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:1749: writing smithi10548823-4 from 3305941 to 3899257 tid 3 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:1733: done (15 left) 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:1734: done (14 left) 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:1750: rollback oid 9 current snap is 37 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 9 to 32 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:1736: finishing write tid 3 to smithi10548823-11 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:1736: finishing write tid 4 to smithi10548823-11 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:1736: finishing write tid 5 to smithi10548823-11 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:1736: finishing write tid 6 to smithi10548823-11 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:1736: oid 11 updating version 0 to 164 2024-04-20T02:52:48.569 INFO:tasks.rados.rados.0.smithi105.stdout:1736: oid 11 version 164 is already newer than 160 2024-04-20T02:52:48.570 INFO:tasks.rados.rados.0.smithi105.stdout:1736: oid 11 version 164 is already newer than 163 2024-04-20T02:52:48.570 INFO:tasks.rados.rados.0.smithi105.stdout:1736: oid 11 version 164 is already newer than 162 2024-04-20T02:52:48.570 INFO:tasks.rados.rados.0.smithi105.stdout:1736: oid 11 version 164 is already newer than 161 2024-04-20T02:52:48.570 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 164 (ObjNum 347 snap 37 seq_num 347) dirty exists 2024-04-20T02:52:48.570 INFO:tasks.rados.rados.0.smithi105.stdout:1736: left oid 11 (ObjNum 347 snap 37 seq_num 347) 2024-04-20T02:52:48.570 INFO:tasks.rados.rados.0.smithi105.stdout:1740: finishing set_chunk to oid 16 2024-04-20T02:52:48.570 INFO:tasks.rados.rados.0.smithi105.stdout:1740: got expected ENOENT (src dne) 2024-04-20T02:52:48.570 INFO:tasks.rados.rados.0.smithi105.stdout:1738: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:52:48.571 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:48.572 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:48.572+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:48.673 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:48.674+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:48.744 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:48.744 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 127.00816851800001, 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:52:48.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:52:48.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:52:48.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:52:48.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.976879457, 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:52:48.750 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.751 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.751 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.751 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.751 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.751 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:52:48.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:52:48.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:52:48.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.972189001, 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.757 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:52:48.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:52:48.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:52:48.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:52:48.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.96543253199999, 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:52:48.763 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.765 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:52:48.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:48.769 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.95561702099999, 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:52:48.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:48.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.775 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.95457308899999, 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.776 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:52:48.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:48.781 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.954551929, 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:48.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:52:48.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:52:48.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.74698134800001, 2024-04-20T02:52:48.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.788 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.789 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:48.790 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.732786703, 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.791 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:52:48.792 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:52:48.793 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.794 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.72681944599999, 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:52:48.795 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:52:48.796 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:52:48.797 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.798 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.718413443, 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.799 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.800 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.801 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.802 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.61478140200001, 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.803 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:52:48.804 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:52:48.805 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:52:48.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.384185092, 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.807 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:52:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:52:48.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.35724094299999, 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.815 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.33188267600001, 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:52:48.819 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.303742232, 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.26637362699999, 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.824 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:48.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:52:48.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:52:48.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:52:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:52:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:52:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.828 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:52:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.266076704, 2024-04-20T02:52:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:52:48.829 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:52:48.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.833 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.260993586, 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:48.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:52:48.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:52:48.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:52:48.837 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 126.233991027, 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:48.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:52:48.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:52:48.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:52:48.841 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.842 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.842 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.842 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:48.842 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:48.842 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:48.845 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:48.845 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:48.844+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:48.846 INFO:tasks.rados.rados.0.smithi105.stdout:1737: finishing set_chunk to oid 39 2024-04-20T02:52:48.846 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 133 (ObjNum 283 snap 27 seq_num 283) dirty exists 2024-04-20T02:52:48.846 INFO:tasks.rados.rados.0.smithi105.stdout:1739: got (0) Success 2024-04-20T02:52:48.846 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 66 (ObjNum 124 snap 9 seq_num 124) dirty exists 2024-04-20T02:52:48.846 INFO:tasks.rados.rados.0.smithi105.stdout:1741: finishing set_chunk to oid 26 2024-04-20T02:52:48.846 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 386 (ObjNum 331 snap 35 seq_num 331) dirty exists 2024-04-20T02:52:48.846 INFO:tasks.rados.rados.0.smithi105.stdout:1742: finishing write tid 1 to smithi10548823-36 2024-04-20T02:52:48.846 INFO:tasks.rados.rados.0.smithi105.stdout:1742: finishing write tid 2 to smithi10548823-36 2024-04-20T02:52:48.846 INFO:tasks.rados.rados.0.smithi105.stdout:1742: finishing write tid 3 to smithi10548823-36 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1742: finishing write tid 4 to smithi10548823-36 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1742: finishing write tid 5 to smithi10548823-36 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1742: oid 36 updating version 0 to 432 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1742: oid 36 version 432 is already newer than 429 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1742: oid 36 version 432 is already newer than 430 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1742: oid 36 version 432 is already newer than 431 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 432 (ObjNum 348 snap 37 seq_num 348) dirty exists 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1742: left oid 36 (ObjNum 348 snap 37 seq_num 348) 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1744: finishing set_chunk to oid 30 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 1056768 length: 1410560 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 30 v 0 (ObjNum 226 snap 22 seq_num 226) dirty exists 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1745: finishing set_chunk to oid 23 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 120 (ObjNum 320 snap 34 seq_num 320) dirty exists 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1736: done (14 left) 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1737: done (13 left) 2024-04-20T02:52:48.847 INFO:tasks.rados.rados.0.smithi105.stdout:1738: done (12 left) 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1739: done (11 left) 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1740: done (10 left) 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1741: done (9 left) 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1742: done (8 left) 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1743: done (7 left) 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1744: done (6 left) 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1745: done (5 left) 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1751: read oid 11 snap -1 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1751: expect (ObjNum 347 snap 37 seq_num 347) 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1752: read oid 32 snap -1 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1752: expect (ObjNum 342 snap 37 seq_num 342) 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1753: tier_evict oid 46 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1753: tier_evict oid 46 snap -1 2024-04-20T02:52:48.848 INFO:tasks.rados.rados.0.smithi105.stdout:1747: expect (ObjNum 292 snap 32 seq_num 292) 2024-04-20T02:52:48.885 INFO:tasks.ceph.mon.a.smithi105.stderr:2024-04-20T02:52:48.885+0000 7f7f0b14c700 -1 mon.a@0(leader).osd e149 definitely_dead 0 2024-04-20T02:52:48.945 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:48.945+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:49.002 INFO:tasks.rados.rados.0.smithi105.stdout:1746: finishing write tid 1 to smithi10548823-47 2024-04-20T02:52:49.002 INFO:tasks.rados.rados.0.smithi105.stdout:1746: finishing write tid 2 to smithi10548823-47 2024-04-20T02:52:49.002 INFO:tasks.rados.rados.0.smithi105.stdout:1747: done (7 left) 2024-04-20T02:52:49.002 INFO:tasks.rados.rados.0.smithi105.stdout:1754: read oid 15 snap -1 2024-04-20T02:52:49.002 INFO:tasks.rados.rados.0.smithi105.stdout:1754: expect deleted 2024-04-20T02:52:49.002 INFO:tasks.rados.rados.0.smithi105.stdout:1755: read oid 28 snap -1 2024-04-20T02:52:49.002 INFO:tasks.rados.rados.0.smithi105.stdout:1755: expect (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:52:49.002 INFO:tasks.rados.rados.0.smithi105.stdout:1756: write oid 49 current snap is 37 2024-04-20T02:52:49.002 INFO:tasks.rados.rados.0.smithi105.stdout:1746: finishing write tid 3 to smithi10548823-47 2024-04-20T02:52:49.002 INFO:tasks.rados.rados.0.smithi105.stdout:1756: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-04-20T02:52:49.009 INFO:tasks.rados.rados.0.smithi105.stdout:1756: writing smithi10548823-49 from 716064 to 1355950 tid 1 2024-04-20T02:52:49.015 INFO:tasks.rados.rados.0.smithi105.stdout:1756: writing smithi10548823-49 from 1815191 to 2315640 tid 2 2024-04-20T02:52:49.015 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:49.016 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:49.016 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:49.022 INFO:tasks.rados.rados.0.smithi105.stdout:1756: writing smithi10548823-49 from 2817011 to 3343936 tid 3 2024-04-20T02:52:49.022 INFO:tasks.rados.rados.0.smithi105.stdout:1757: read oid 6 snap -1 2024-04-20T02:52:49.022 INFO:tasks.rados.rados.0.smithi105.stdout:1757: expect (ObjNum 321 snap 35 seq_num 321) 2024-04-20T02:52:49.022 INFO:tasks.rados.rados.0.smithi105.stdout:1758: delete oid 30 current snap is 37 2024-04-20T02:52:49.022 INFO:tasks.rados.rados.0.smithi105.stdout:1746: finishing write tid 4 to smithi10548823-47 2024-04-20T02:52:49.022 INFO:tasks.rados.rados.0.smithi105.stdout:1746: finishing write tid 5 to smithi10548823-47 2024-04-20T02:52:49.022 INFO:tasks.rados.rados.0.smithi105.stdout:1746: oid 47 updating version 0 to 149 2024-04-20T02:52:49.022 INFO:tasks.rados.rados.0.smithi105.stdout:1746: oid 47 version 149 is already newer than 147 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1746: oid 47 version 149 is already newer than 148 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1746: oid 47 version 149 is already newer than 146 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 149 (ObjNum 349 snap 37 seq_num 349) dirty exists 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1746: left oid 47 (ObjNum 349 snap 37 seq_num 349) 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1748: finishing write tid 1 to smithi10548823-50 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1748: finishing write tid 2 to smithi10548823-50 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1748: finishing write tid 3 to smithi10548823-50 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1748: finishing write tid 4 to smithi10548823-50 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1748: finishing write tid 5 to smithi10548823-50 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1748: oid 50 updating version 0 to 103 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1748: oid 50 updating version 103 to 106 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1748: oid 50 version 106 is already newer than 104 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1748: oid 50 version 106 is already newer than 105 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 50 v 106 (ObjNum 350 snap 37 seq_num 350) dirty exists 2024-04-20T02:52:49.023 INFO:tasks.rados.rados.0.smithi105.stdout:1748: left oid 50 (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:52:49.024 INFO:tasks.rados.rados.0.smithi105.stdout:1753: got (22) Invalid argument 2024-04-20T02:52:49.024 INFO:tasks.rados.rados.0.smithi105.stdout:1751: expect (ObjNum 347 snap 37 seq_num 347) 2024-04-20T02:52:49.027 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:49.071 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:49.072+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:49.146 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:49.146+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:49.156 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:49.156 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:49.156 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:49.156 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:49.156 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:49.167 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:49.250 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:49.251+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:49.258 INFO:tasks.rados.rados.0.smithi105.stdout:1757: expect (ObjNum 321 snap 35 seq_num 321) 2024-04-20T02:52:49.337 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:49.337 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:49.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:49.337 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:49.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.337 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 99.887227457999998, 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:52:49.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.837044691000003, 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.342 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:52:49.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:52:49.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:52:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:52:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:52:49.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:52:49.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.347 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.347 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:49.348 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:49.348 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.836979241999998, 2024-04-20T02:52:49.348 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:52:49.348 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.348 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:49.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.349 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.349 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.349 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:52:49.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.349 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:49.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:49.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:52:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:52:49.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:52:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:52:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:52:49.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:52:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:52:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:49.356 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 90.836953743999999, 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:52:49.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:52:49.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:52:49.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2524 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1723287~549075] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.049444260999998, 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054106970999999997, 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.361 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2524 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286419+0000", 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286924+0000", 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050494600000000004 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286944+0000", 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9833000000000001e-05 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286957+0000", 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2901000000000001e-05 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298074+0000", 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011117320999999999 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298151+0000", 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6909999999999994e-05 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:49.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298274+0000", 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012279700000000001 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326267+0000", 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027993284 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331226+0000", 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049590559999999999 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340442+0000", 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092155740000000007 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340488+0000", 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6208000000000002e-05 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.366 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340528+0000", 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0277000000000001e-05 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2525 4.8 4:1afa5d2b:::smithi10548823-17:head [write 2781837~703343] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.042693047, 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050020689, 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.367 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2525 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:49.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293172+0000", 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293838+0000", 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066662700000000004 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293857+0000", 2024-04-20T02:52:49.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.929e-05 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293867+0000", 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9369999999999999e-06 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310149+0000", 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016281713999999999 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310257+0000", 2024-04-20T02:52:49.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108262 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310439+0000", 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018146300000000001 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.327814+0000", 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017375043 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331934+0000", 2024-04-20T02:52:49.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041200489999999998 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343112+0000", 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011177991999999999 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343146+0000", 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3813000000000003e-05 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343193+0000", 2024-04-20T02:52:49.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2526 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (57) in=7b,truncate 3630312] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.293877+0000", 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.041988473000004, 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049424889, 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.373 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2526 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293877+0000", 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293877+0000", 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293876+0000", 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293881+0000", 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.152e-06 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293891+0000", 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5529999999999998e-06 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293900+0000", 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3449999999999995e-06 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.325730+0000", 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031829673000000003 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.325823+0000", 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2812000000000005e-05 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.325929+0000", 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106263 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.327842+0000", 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019130060000000001 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331975+0000", 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041330780000000001 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343226+0000", 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011250657000000001 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343260+0000", 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4025e-05 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343302+0000", 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2199e-05 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.378 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2527 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 61.041960177, 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049568319, 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2527 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293905+0000", 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293908+0000", 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.605e-06 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293915+0000", 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9589999999999998e-06 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293919+0000", 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6909999999999996e-06 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326286+0000", 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032366250999999999 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326340+0000", 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4898000000000002e-05 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343324+0000", 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016983682 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343400+0000", 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.5674999999999996e-05 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343474+0000", 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3948000000000004e-05 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.383 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3637 4.1 e137/60 4:80c7a726:::smithi10548823-19:head v 137'124, mlcod=120'120)", 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.711456702, 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052640605, 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624408+0000", 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624537+0000", 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012880199999999999 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624548+0000", 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0216e-05 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624556+0000", 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6179999999999995e-06 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624603+0000", 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6601999999999999e-05 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624632+0000", 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9277000000000001e-05 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.676999+0000", 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052366639999999999 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.677049+0000", 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0822999999999998e-05 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.387 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3880 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'231, mlcod=141'228)", 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.9289617220000004, 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.061291667000000001, 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:52:49.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406903+0000", 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407359+0000", 2024-04-20T02:52:49.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00045623700000000002 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407370+0000", 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1052e-05 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407377+0000", 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1080000000000004e-06 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407429+0000", 2024-04-20T02:52:49.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2033999999999997e-05 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407458+0000", 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8050000000000001e-05 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468161+0000", 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060703628000000003 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468196+0000", 2024-04-20T02:52:49.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4344999999999997e-05 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3881 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'232, mlcod=141'228)", 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.418645+0000", 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.917220725, 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.049593834000000003, 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.392 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418645+0000", 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418645+0000", 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418643+0000", 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.420175+0000", 2024-04-20T02:52:49.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001531803 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.420187+0000", 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2218999999999999e-05 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.420197+0000", 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.1009999999999998e-06 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.423553+0000", 2024-04-20T02:52:49.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033568560000000001 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.423581+0000", 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7338000000000002e-05 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468214+0000", 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044633284000000002 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468239+0000", 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4689e-05 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3888 4.7 4:e2c0d65b:::smithi10548823-46:head [write 413538~536345,stat] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.9104085939999997, 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.059175945000000001, 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.396 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3888 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425456+0000", 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426097+0000", 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00064115099999999996 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426113+0000", 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6118000000000001e-05 2024-04-20T02:52:49.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426122+0000", 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.115e-06 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426179+0000", 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6815999999999999e-05 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426316+0000", 2024-04-20T02:52:49.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000136606 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426499+0000", 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000183191 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443652+0000", 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017153471999999999 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484036+0000", 2024-04-20T02:52:49.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040384097000000001 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484231+0000", 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000194811 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484363+0000", 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013146300000000001 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484426+0000", 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3563000000000001e-05 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484633+0000", 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020662299999999999 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3889 4.7 4:e2c0d65b:::smithi10548823-46:head [write 1395221~583781] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:52:49.402 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.9035146190000001, 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.057918795000000002, 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3889 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432350+0000", 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433490+0000", 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001140606 2024-04-20T02:52:49.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433506+0000", 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5828000000000002e-05 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433515+0000", 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8279999999999992e-06 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443692+0000", 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010177222 2024-04-20T02:52:49.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443839+0000", 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014662 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.444072+0000", 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023362200000000001 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484114+0000", 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040041641000000003 2024-04-20T02:52:49.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484244+0000", 2024-04-20T02:52:49.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012987200000000001 2024-04-20T02:52:49.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484279+0000", 2024-04-20T02:52:49.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5301e-05 2024-04-20T02:52:49.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490055+0000", 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057763459999999999 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490115+0000", 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9320999999999997e-05 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490270+0000", 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015498199999999999 2024-04-20T02:52:49.408 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3890 4.7 4:e2c0d65b:::smithi10548823-46:head [write 2556103~658617] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.8955005180000004, 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066878325000000002, 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.409 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3890 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440363+0000", 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441831+0000", 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001467632 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441842+0000", 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0540000000000001e-05 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441850+0000", 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7260000000000008e-06 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463536+0000", 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02168544 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463681+0000", 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000145086 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463941+0000", 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00025968600000000002 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484317+0000", 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020376452 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490318+0000", 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060012490000000002 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490398+0000", 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9598000000000006e-05 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506988+0000", 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016589706999999999 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507066+0000", 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8299000000000004e-05 2024-04-20T02:52:49.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507243+0000", 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000177503 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3891 4.7 4:e2c0d65b:::smithi10548823-46:head [setxattr _header (57) in=7b,truncate 3613538] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.8940050089999998, 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088590031, 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.415 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3891 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:52:49.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441859+0000", 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441864+0000", 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8069999999999999e-06 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441869+0000", 2024-04-20T02:52:49.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0989999999999998e-06 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441875+0000", 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5820000000000001e-06 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484675+0000", 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042799867999999998 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484817+0000", 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014236899999999999 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485048+0000", 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023057199999999999 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490359+0000", 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053116819999999999 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506963+0000", 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016603562999999998 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507256+0000", 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00029288699999999998 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530316+0000", 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023060343000000001 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530356+0000", 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9301999999999997e-05 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530451+0000", 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5037999999999994e-05 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3892 4.7 4:e2c0d65b:::smithi10548823-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.8939853209999997, 2024-04-20T02:52:49.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088912482000000001, 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3892 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441883+0000", 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9579999999999999e-06 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441889+0000", 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7499999999999997e-06 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441895+0000", 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485431+0000", 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043536321000000003 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485537+0000", 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010612599999999999 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530489+0000", 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044951996000000001 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530598+0000", 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108937 2024-04-20T02:52:49.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530793+0000", 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019441 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3898 4.8 4:1c68e542:::smithi10548823-8:head [write 447152~403196] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.869642238, 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.056917473000000003, 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.426 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3898 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:52:49.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466222+0000", 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466722+0000", 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00049941299999999998 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466734+0000", 2024-04-20T02:52:49.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2860999999999999e-05 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466743+0000", 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7159999999999992e-06 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466809+0000", 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6165000000000001e-05 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466928+0000", 2024-04-20T02:52:49.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011869199999999999 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.467158+0000", 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000230251 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498338+0000", 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.03117996 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.522958+0000", 2024-04-20T02:52:49.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024619318000000001 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523002+0000", 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4675999999999997e-05 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523082+0000", 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9803999999999997e-05 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523141+0000", 2024-04-20T02:52:49.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8767000000000002e-05 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3899 4.8 4:1c68e542:::smithi10548823-8:head [write 1512185~587450] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.861714471, 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060518556000000001, 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3899 2024-04-20T02:52:49.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474149+0000", 2024-04-20T02:52:49.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474686+0000", 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053696199999999999 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474703+0000", 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.615e-05 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474713+0000", 2024-04-20T02:52:49.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0829000000000001e-05 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478533+0000", 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003819687 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478640+0000", 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106974 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:49.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478766+0000", 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125859 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523803+0000", 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045036616000000002 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526432+0000", 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026291489999999999 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534589+0000", 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081572209999999992 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534623+0000", 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534670+0000", 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6968999999999997e-05 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4016 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 665643~697786] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.0745457060000001, 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060676251, 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4016 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261318+0000", 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262081+0000", 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076285299999999997 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262095+0000", 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3986e-05 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262118+0000", 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2407000000000001e-05 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262164+0000", 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6248999999999997e-05 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262238+0000", 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4202000000000005e-05 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262363+0000", 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012437799999999999 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314944+0000", 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052581687000000002 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317407+0000", 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024628300000000001 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321887+0000", 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0044803020000000002 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321941+0000", 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3769e-05 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321996+0000", 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4988000000000003e-05 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4017 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 2039745~753122] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:52:49.443 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 1.0669159079999999, 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053146383999999998, 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4017 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268948+0000", 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269678+0000", 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072975799999999995 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269693+0000", 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.453e-05 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269704+0000", 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1623e-05 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277190+0000", 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074859849999999997 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277276+0000", 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5809999999999993e-05 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277405+0000", 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012914600000000001 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314952+0000", 2024-04-20T02:52:49.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037546931999999998 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317441+0000", 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024889840000000001 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322031+0000", 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004589465 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322059+0000", 2024-04-20T02:52:49.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322096+0000", 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591999999999998e-05 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:49.449 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:49.451 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1756: finishing write tid 1 to smithi10548823-49 2024-04-20T02:52:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1756: finishing write tid 2 to smithi10548823-49 2024-04-20T02:52:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1756: finishing write tid 3 to smithi10548823-49 2024-04-20T02:52:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1756: finishing write tid 4 to smithi10548823-49 2024-04-20T02:52:49.452 INFO:tasks.rados.rados.0.smithi105.stdout:1756: finishing write tid 5 to smithi10548823-49 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1756: oid 49 updating version 0 to 240 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1756: oid 49 updating version 240 to 242 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1756: oid 49 version 242 is already newer than 241 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1756: oid 49 updating version 242 to 243 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 243 (ObjNum 352 snap 37 seq_num 352) dirty exists 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1756: left oid 49 (ObjNum 352 snap 37 seq_num 352) 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1746: done (11 left) 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1748: done (10 left) 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1751: done (9 left) 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1753: done (8 left) 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1756: done (7 left) 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1757: done (6 left) 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1758: done (5 left) 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1759: write (excl) oid 6 current snap is 37 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1759: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2024-04-20T02:52:49.453 INFO:tasks.rados.rados.0.smithi105.stdout:1759: writing smithi10548823-6 from 749678 to 1540062 tid 1 2024-04-20T02:52:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:1759: writing smithi10548823-6 from 2332155 to 2836273 tid 2 2024-04-20T02:52:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:1759: writing smithi10548823-6 from 3271857 to 3821326 tid 3 2024-04-20T02:52:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:1760: write oid 24 current snap is 37 2024-04-20T02:52:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:1760: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-04-20T02:52:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:1760: writing smithi10548823-24 from 766485 to 1232118 tid 1 2024-04-20T02:52:49.454 INFO:tasks.rados.rados.0.smithi105.stdout:1760: writing smithi10548823-24 from 1990637 to 2638413 tid 2 2024-04-20T02:52:49.454 INFO:teuthology.orchestra.run.smithi105.stderr:marked down osd.4. 2024-04-20T02:52:49.454 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:52:49.455 INFO:tasks.rados.rados.0.smithi105.stdout:1760: writing smithi10548823-24 from 3299280 to 3966485 tid 3 2024-04-20T02:52:49.455 INFO:tasks.rados.rados.0.smithi105.stdout:1761: write oid 44 current snap is 37 2024-04-20T02:52:49.455 INFO:tasks.rados.rados.0.smithi105.stdout:1761: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-04-20T02:52:49.455 INFO:tasks.rados.rados.0.smithi105.stdout:1761: writing smithi10548823-44 from 783292 to 1324174 tid 1 2024-04-20T02:52:49.457 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 3, 4, 6, 7, 5] 2024-04-20T02:52:49.458 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-04-20T02:52:49.459 INFO:tasks.rados.rados.0.smithi105.stdout:1761: writing smithi10548823-44 from 1932766 to 2440553 tid 2 2024-04-20T02:52:49.459 INFO:tasks.rados.rados.0.smithi105.stdout:1761: writing smithi10548823-44 from 3210350 to 3782363 tid 3 2024-04-20T02:52:49.459 INFO:tasks.rados.rados.0.smithi105.stdout:1762: tier_evict oid 23 2024-04-20T02:52:49.459 INFO:tasks.rados.rados.0.smithi105.stdout:1762: tier_evict oid 23 snap -1 2024-04-20T02:52:49.459 INFO:tasks.rados.rados.0.smithi105.stdout:1763: write oid 5 current snap is 37 2024-04-20T02:52:49.459 INFO:tasks.rados.rados.0.smithi105.stdout:1763: seq_num 357 ranges {400099=616131,1591248=408851} 2024-04-20T02:52:49.459 INFO:tasks.rados.rados.0.smithi105.stdout:1763: writing smithi10548823-5 from 400099 to 1016230 tid 1 2024-04-20T02:52:49.459 INFO:tasks.rados.rados.0.smithi105.stdout:1763: writing smithi10548823-5 from 1591248 to 2000099 tid 2 2024-04-20T02:52:49.459 INFO:tasks.rados.rados.0.smithi105.stdout:1764: snap_remove snap 36 2024-04-20T02:52:49.459 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:49.451+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:49.544 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:49.551 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:49.552+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:49.634 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:49.634 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:49.635 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:49.635 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:49.646 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:49.690 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:49.690+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:49.697 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:49.717 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:49.737 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:49.753 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:49.754+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:49.757 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:49.777 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:49.784 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:49.784 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:49.794 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:49.797 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:49.820 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:49.820 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:49.820 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:49.820 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:49.820 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:49.820 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-20T02:52:49.820 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:49.820 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:49.820 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:49.823 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:49.859 INFO:tasks.ceph.mon.a.smithi105.stderr:2024-04-20T02:52:49.859+0000 7f7f0b14c700 -1 mon.a@0(leader).osd e150 definitely_dead 0 2024-04-20T02:52:49.860 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:49.860+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:49.927 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:49.927 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:49.937 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:49.979 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:49.980+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:50.094 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:50.056+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:50.106 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:50.107 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:50.119 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:50.163 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:50.163+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:50.258 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:50.257+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:50.297 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:50.298 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:50.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:50.359 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:50.359+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:50.404 INFO:tasks.rados.rados.0.smithi105.stdout:1764: done (10 left) 2024-04-20T02:52:50.404 INFO:tasks.rados.rados.0.smithi105.stdout:1765: write (excl) oid 21 current snap is 37 2024-04-20T02:52:50.404 INFO:tasks.rados.rados.0.smithi105.stdout:1765: seq_num 358 ranges {416906=407733,1366083=511456} 2024-04-20T02:52:50.410 INFO:tasks.rados.rados.0.smithi105.stdout:1765: writing smithi10548823-21 from 416906 to 824639 tid 1 2024-04-20T02:52:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:1765: writing smithi10548823-21 from 1366083 to 1877539 tid 2 2024-04-20T02:52:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:1766: read oid 41 snap -1 2024-04-20T02:52:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:1766: expect (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:52:50.413 INFO:tasks.rados.rados.0.smithi105.stdout:1767: set_chunk oid 14 target oid 2024-04-20T02:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:marked out osd.4. 2024-04-20T02:52:50.436 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-04-20T02:52:50.437 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:52:50.437 DEBUG:teuthology.orchestra.run.smithi138:> type ceph-objectstore-tool 2024-04-20T02:52:50.454 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-20T02:52:50.454 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-04-20T02:52:50.470 INFO:tasks.rados.rados.0.smithi105.stdout:1767: set_chunk oid 14 offset: 272384 length: 1132032 target oid a7fba1f52b6da7bcda43d48ae1481b236217daab845625381b62aefe18e6b1d2 offset: 0 snap 35 2024-04-20T02:52:50.471 INFO:tasks.rados.rados.0.smithi105.stdout:1768: write oid 7 current snap is 37 2024-04-20T02:52:50.471 INFO:tasks.rados.rados.0.smithi105.stdout:1762: got (22) Invalid argument 2024-04-20T02:52:50.471 INFO:tasks.rados.rados.0.smithi105.stdout:1768: seq_num 359 ranges {433713=482982,1424565=609148} 2024-04-20T02:52:50.476 INFO:tasks.rados.rados.0.smithi105.stdout:1768: writing smithi10548823-7 from 433713 to 916695 tid 1 2024-04-20T02:52:50.484 INFO:tasks.rados.rados.0.smithi105.stdout:1768: writing smithi10548823-7 from 1424565 to 2033713 tid 2 2024-04-20T02:52:50.484 INFO:tasks.rados.rados.0.smithi105.stdout:1762: done (13 left) 2024-04-20T02:52:50.484 INFO:tasks.rados.rados.0.smithi105.stdout:1769: read oid 25 snap -1 2024-04-20T02:52:50.484 INFO:tasks.rados.rados.0.smithi105.stdout:1769: expect (ObjNum 294 snap 32 seq_num 294) 2024-04-20T02:52:50.484 INFO:tasks.rados.rados.0.smithi105.stdout:1770: write oid 36 current snap is 37 2024-04-20T02:52:50.484 INFO:tasks.rados.rados.0.smithi105.stdout:1749: finishing write tid 1 to smithi10548823-4 2024-04-20T02:52:50.484 INFO:tasks.rados.rados.0.smithi105.stdout:1749: finishing write tid 2 to smithi10548823-4 2024-04-20T02:52:50.484 INFO:tasks.rados.rados.0.smithi105.stdout:1749: finishing write tid 3 to smithi10548823-4 2024-04-20T02:52:50.484 INFO:tasks.rados.rados.0.smithi105.stdout:1770: seq_num 360 ranges {450520=558231,1483047=515125} 2024-04-20T02:52:50.489 INFO:tasks.rados.rados.0.smithi105.stdout:1770: writing smithi10548823-36 from 450520 to 1008751 tid 1 2024-04-20T02:52:50.495 INFO:tasks.rados.rados.0.smithi105.stdout:1770: writing smithi10548823-36 from 1483047 to 1998172 tid 2 2024-04-20T02:52:50.495 INFO:tasks.rados.rados.0.smithi105.stdout:1771: read oid 30 snap -1 2024-04-20T02:52:50.495 INFO:tasks.rados.rados.0.smithi105.stdout:1771: expect deleted 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:1749: finishing write tid 4 to smithi10548823-4 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:1749: finishing write tid 5 to smithi10548823-4 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:1749: oid 4 updating version 0 to 252 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:1749: oid 4 updating version 252 to 255 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:1749: oid 4 version 255 is already newer than 254 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:1749: oid 4 version 255 is already newer than 253 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 255 (ObjNum 351 snap 37 seq_num 351) dirty exists 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:1749: left oid 4 (ObjNum 351 snap 37 seq_num 351) 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:1750: finishing rollback tid 1 to smithi10548823-9 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-20T02:52:50.496 INFO:tasks.rados.rados.0.smithi105.stdout:1766: expect (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:52:50.497 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:50.496+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:50.511 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:50.511 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:50.522 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:50.560 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:50.561+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:50.629 INFO:tasks.rados.rados.0.smithi105.stdout:1765: finishing write tid 1 to smithi10548823-21 2024-04-20T02:52:50.629 INFO:tasks.rados.rados.0.smithi105.stdout:1765: finishing write tid 2 to smithi10548823-21 2024-04-20T02:52:50.629 INFO:tasks.rados.rados.0.smithi105.stdout:1749: done (15 left) 2024-04-20T02:52:50.629 INFO:tasks.rados.rados.0.smithi105.stdout:1750: done (14 left) 2024-04-20T02:52:50.629 INFO:tasks.rados.rados.0.smithi105.stdout:1766: done (13 left) 2024-04-20T02:52:50.629 INFO:tasks.rados.rados.0.smithi105.stdout:1772: write (excl) oid 22 current snap is 37 2024-04-20T02:52:50.629 INFO:tasks.rados.rados.0.smithi105.stdout:1765: finishing write tid 3 to smithi10548823-21 2024-04-20T02:52:50.629 INFO:tasks.rados.rados.0.smithi105.stdout:1772: seq_num 361 ranges {467327=633480,1825176=242151} 2024-04-20T02:52:50.634 INFO:tasks.rados.rados.0.smithi105.stdout:1772: writing smithi10548823-22 from 467327 to 1100807 tid 1 2024-04-20T02:52:50.640 INFO:tasks.rados.rados.0.smithi105.stdout:1772: writing smithi10548823-22 from 1825176 to 2067327 tid 2 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1773: write oid 18 current snap is 37 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1765: finishing write tid 4 to smithi10548823-21 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1765: oid 21 updating version 0 to 167 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1765: oid 21 updating version 167 to 168 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1765: oid 21 updating version 168 to 169 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 169 (ObjNum 358 snap 37 seq_num 358) dirty exists 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1765: left oid 21 (ObjNum 358 snap 37 seq_num 358) 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1767: finishing set_chunk to oid 14 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 14 set_chunk returned error code -95 offset: 272384 length: 1132032 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 0 (ObjNum 343 snap 37 seq_num 343) dirty exists 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1768: finishing write tid 1 to smithi10548823-7 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1768: finishing write tid 2 to smithi10548823-7 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1768: finishing write tid 3 to smithi10548823-7 2024-04-20T02:52:50.641 INFO:tasks.rados.rados.0.smithi105.stdout:1768: finishing write tid 4 to smithi10548823-7 2024-04-20T02:52:50.642 INFO:tasks.rados.rados.0.smithi105.stdout:1768: oid 7 updating version 0 to 241 2024-04-20T02:52:50.642 INFO:tasks.rados.rados.0.smithi105.stdout:1768: oid 7 version 241 is already newer than 240 2024-04-20T02:52:50.642 INFO:tasks.rados.rados.0.smithi105.stdout:1768: oid 7 updating version 241 to 242 2024-04-20T02:52:50.642 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 242 (ObjNum 359 snap 37 seq_num 359) dirty exists 2024-04-20T02:52:50.642 INFO:tasks.rados.rados.0.smithi105.stdout:1768: left oid 7 (ObjNum 359 snap 37 seq_num 359) 2024-04-20T02:52:50.642 INFO:tasks.rados.rados.0.smithi105.stdout:1773: seq_num 362 ranges {484134=708729,1883658=200476} 2024-04-20T02:52:50.648 INFO:tasks.rados.rados.0.smithi105.stdout:1773: writing smithi10548823-18 from 484134 to 1192863 tid 1 2024-04-20T02:52:50.654 INFO:tasks.rados.rados.0.smithi105.stdout:1773: writing smithi10548823-18 from 1883658 to 2084134 tid 2 2024-04-20T02:52:50.654 INFO:tasks.rados.rados.0.smithi105.stdout:1765: done (14 left) 2024-04-20T02:52:50.654 INFO:tasks.rados.rados.0.smithi105.stdout:1767: done (13 left) 2024-04-20T02:52:50.654 INFO:tasks.rados.rados.0.smithi105.stdout:1768: done (12 left) 2024-04-20T02:52:50.654 INFO:tasks.rados.rados.0.smithi105.stdout:1771: done (11 left) 2024-04-20T02:52:50.654 INFO:tasks.rados.rados.0.smithi105.stdout:1774: set_chunk oid 8 target oid 2024-04-20T02:52:50.661 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:50.663+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1774: set_chunk oid 8 offset: 2549760 length: 457216 target oid ae4c3a2239b98cabbac2917d2d234db9a65a1e9186a7c6d1482bdbea56659fb0 offset: 0 snap -1 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1775: tier_flush oid 45 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1775: tier_flush oid 45 snap -1 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1772: finishing write tid 1 to smithi10548823-22 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1772: finishing write tid 2 to smithi10548823-22 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1772: finishing write tid 3 to smithi10548823-22 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1772: finishing write tid 4 to smithi10548823-22 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1772: oid 22 updating version 0 to 125 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1772: oid 22 updating version 125 to 126 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1772: oid 22 version 126 is already newer than 124 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 126 (ObjNum 361 snap 37 seq_num 361) dirty exists 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1772: left oid 22 (ObjNum 361 snap 37 seq_num 361) 2024-04-20T02:52:50.671 INFO:tasks.rados.rados.0.smithi105.stdout:1772: done (12 left) 2024-04-20T02:52:50.672 INFO:tasks.rados.rados.0.smithi105.stdout:1776: delete oid 30 current snap is 37 2024-04-20T02:52:50.672 INFO:tasks.rados.rados.0.smithi105.stdout:1775: got (16) Device or resource busy 2024-04-20T02:52:50.672 INFO:tasks.rados.rados.0.smithi105.stdout:check_oldest_snap_flushed oid 45 v 227 (ObjNum 325 snap 35 seq_num 325) unflushed 35 2024-04-20T02:52:50.673 INFO:tasks.rados.rados.0.smithi105.stdout:1775: done (12 left) 2024-04-20T02:52:50.673 INFO:tasks.rados.rados.0.smithi105.stdout:1776: done (11 left) 2024-04-20T02:52:50.673 INFO:tasks.rados.rados.0.smithi105.stdout:1777: read oid 1 snap -1 2024-04-20T02:52:50.673 INFO:tasks.rados.rados.0.smithi105.stdout:1777: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:50.673 INFO:tasks.rados.rados.0.smithi105.stdout:1778: rollback oid 34 current snap is 37 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 34 to 35 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:1779: tier_evict oid 42 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:1779: tier_evict oid 42 snap -1 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:1780: tier_evict oid 45 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:1780: tier_evict oid 45 snap -1 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:1781: read oid 35 snap 32 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:1781: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:1780: got (22) Invalid argument 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:1780: done (15 left) 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:1782: read oid 27 snap 35 2024-04-20T02:52:50.674 INFO:tasks.rados.rados.0.smithi105.stdout:1782: expect (ObjNum 317 snap 34 seq_num 317) 2024-04-20T02:52:50.675 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:50.676 INFO:tasks.rados.rados.0.smithi105.stdout:1777: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:50.722 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:50.722 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:50.735 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:50.763 INFO:tasks.rados.rados.0.smithi105.stdout:1781: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:52:50.764 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:50.764+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:50.839 INFO:tasks.rados.rados.0.smithi105.stdout:1774: finishing set_chunk to oid 8 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 136 (ObjNum 336 snap 36 seq_num 336) dirty exists 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1778: finishing rollback tid 0 to smithi10548823-34 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1778: finishing rollback tid 1 to smithi10548823-34 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1774: done (15 left) 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1777: done (14 left) 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1781: done (13 left) 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1783: tier_evict oid 49 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1783: tier_evict oid 49 snap -1 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1778: finishing rollback tid 2 to smithi10548823-34 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 260 (ObjNum 319 snap 34 seq_num 319) dirty exists 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1778: done (13 left) 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1784: read oid 23 snap -1 2024-04-20T02:52:50.840 INFO:tasks.rados.rados.0.smithi105.stdout:1784: expect (ObjNum 320 snap 34 seq_num 320) 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:1785: read oid 21 snap -1 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:1785: expect (ObjNum 358 snap 37 seq_num 358) 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:1786: read oid 47 snap 35 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:1786: expect deleted 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:1783: got (22) Invalid argument 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:1783: done (15 left) 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:1787: read oid 50 snap -1 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:1787: expect (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:1786: done (15 left) 2024-04-20T02:52:50.841 INFO:tasks.rados.rados.0.smithi105.stdout:1788: set_chunk oid 4 target oid 2024-04-20T02:52:50.863 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:50.864+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:50.890 INFO:tasks.rados.rados.0.smithi105.stdout:1788: set_chunk oid 4 offset: 2250752 length: 1549824 target oid 4cf2db757b9d593be5b534135b5682117e8a57e7fa6732ce95143672f161e715 offset: 0 snap -1 2024-04-20T02:52:50.891 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:50.891 INFO:tasks.rados.rados.0.smithi105.stdout:1785: expect (ObjNum 358 snap 37 seq_num 358) 2024-04-20T02:52:50.900 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:50.900 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:50.910 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:50.964 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:50.966+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:51.005 INFO:tasks.rados.rados.0.smithi105.stdout:1784: expect (ObjNum 320 snap 34 seq_num 320) 2024-04-20T02:52:51.040 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:51.041 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:51.051 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:51.065 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:51.066+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:51.166 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:51.167+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:51.231 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:51.231 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:51.242 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:51.248 INFO:tasks.rados.rados.0.smithi105.stdout:1788: finishing set_chunk to oid 4 2024-04-20T02:52:51.249 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 255 (ObjNum 351 snap 37 seq_num 351) dirty exists 2024-04-20T02:52:51.249 INFO:tasks.rados.rados.0.smithi105.stdout:1784: done (15 left) 2024-04-20T02:52:51.249 INFO:tasks.rados.rados.0.smithi105.stdout:1785: done (14 left) 2024-04-20T02:52:51.249 INFO:tasks.rados.rados.0.smithi105.stdout:1788: done (13 left) 2024-04-20T02:52:51.249 INFO:tasks.rados.rados.0.smithi105.stdout:1789: set_chunk oid 20 target oid 2024-04-20T02:52:51.266 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:51.267+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:51.367 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:51.368+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:51.376 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:51.376 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:51.386 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:51.457 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:52:51.501 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:51.502+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:51.517 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:51.517 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:51.528 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:51.572 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:51.573+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:51.661 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:51.661 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:51.670 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:51.671+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:51.672 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:51.770 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:51.772+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:51.824 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:51.841 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:51.841 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:51.853 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:51.871 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:51.872+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:51.969 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:51.972 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:51.973+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:51.988 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:51.992 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:51.992 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:51.997 INFO:teuthology.orchestra.run.smithi138.stdout:4.f 2024-04-20T02:52:51.997 INFO:teuthology.orchestra.run.smithi138.stdout:4.e 2024-04-20T02:52:51.997 INFO:teuthology.orchestra.run.smithi138.stdout:4.c 2024-04-20T02:52:51.997 INFO:teuthology.orchestra.run.smithi138.stdout:4.b 2024-04-20T02:52:51.997 INFO:teuthology.orchestra.run.smithi138.stdout:4.5 2024-04-20T02:52:51.997 INFO:teuthology.orchestra.run.smithi138.stdout:4.2 2024-04-20T02:52:51.997 INFO:teuthology.orchestra.run.smithi138.stdout:4.1 2024-04-20T02:52:51.997 INFO:teuthology.orchestra.run.smithi138.stdout:4.13 2024-04-20T02:52:51.998 INFO:teuthology.orchestra.run.smithi138.stdout:4.12 2024-04-20T02:52:51.998 INFO:teuthology.orchestra.run.smithi138.stdout:4.11 2024-04-20T02:52:51.998 INFO:teuthology.orchestra.run.smithi138.stdout:4.9 2024-04-20T02:52:51.998 INFO:teuthology.orchestra.run.smithi138.stdout:3.3 2024-04-20T02:52:51.998 INFO:teuthology.orchestra.run.smithi138.stdout:3.1 2024-04-20T02:52:51.998 INFO:teuthology.orchestra.run.smithi138.stdout:2.6 2024-04-20T02:52:51.998 INFO:teuthology.orchestra.run.smithi138.stdout:2.5 2024-04-20T02:52:52.003 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:52.006 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:52.025 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:52.045 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:52.064 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:52.073 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:52.074+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:52.083 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:52.083 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:52.083 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:52.083 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:52.084 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:52.084 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-20T02:52:52.084 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:52.084 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:52.084 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:52.086 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:52.134 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:52.134 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:52.134 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:52.134 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:52.149 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:52:52.150 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:52.175 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:52.176+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:52.247 INFO:tasks.rados.rados.0.smithi105.stdout:1789: set_chunk oid 20 offset: 1438720 length: 1129472 target oid 55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22 offset: 0 snap 32 2024-04-20T02:52:52.247 INFO:tasks.rados.rados.0.smithi105.stdout:1790: write (excl) oid 29 current snap is 37 2024-04-20T02:52:52.247 INFO:tasks.rados.rados.0.smithi105.stdout:1790: seq_num 364 ranges {517748=459227,1600622=406110} 2024-04-20T02:52:52.252 INFO:tasks.rados.rados.0.smithi105.stdout:1790: writing smithi10548823-29 from 517748 to 976975 tid 1 2024-04-20T02:52:52.259 INFO:tasks.rados.rados.0.smithi105.stdout:1790: writing smithi10548823-29 from 1600622 to 2006732 tid 2 2024-04-20T02:52:52.259 INFO:tasks.rados.rados.0.smithi105.stdout:1791: read oid 45 snap 35 2024-04-20T02:52:52.259 INFO:tasks.rados.rados.0.smithi105.stdout:1791: expect (ObjNum 325 snap 35 seq_num 325) 2024-04-20T02:52:52.259 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:52.259 INFO:tasks.rados.rados.0.smithi105.stdout:1787: expect (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:52:52.276 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:52.277+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:52.324 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:52.324 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:52.324 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:52.324 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:52.324 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:52.335 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:52.379 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:52.380+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:52.466 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 152.862047676, 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:52:52.467 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:52:52.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:52:52.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:52:52.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:52:52.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:52:52.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:52.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 130.98487823599999, 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.474 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:52:52.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:52:52.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:52:52.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:52.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 130.62509258399999, 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.480 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:52.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:52:52.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 130.61472897900001, 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:169 4.0 e29/22 4:0b0a7da5:::smithi10548823-43:head v 29'13, mlcod=29'11)", 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 130.60664449999999, 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442791000000003, 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.489 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859318+0000", 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859317+0000", 2024-04-20T02:52:52.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859805+0000", 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00048811500000000001 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859815+0000", 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0549e-05 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859821+0000", 2024-04-20T02:52:52.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0769999999999996e-06 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859867+0000", 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5550999999999998e-05 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859892+0000", 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5238000000000001e-05 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896724+0000", 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036831383000000002 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.896760+0000", 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6828e-05 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:52.493 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.325086275, 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:52:52.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:52.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.319949165, 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.500 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:52:52.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:52:52.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.505 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.319676153, 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.506 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:52:52.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:52:52.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:52:52.511 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 112.319646768, 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:52.512 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.516 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 89.632319468999995, 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:52:52.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:52:52.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 64.184839246999999, 2024-04-20T02:52:52.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:52:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:52:52.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.525 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 64.167570812999998, 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 40.994425018000001, 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3881 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'232, mlcod=141'228)", 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.0473024780000006, 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050172267999999999, 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418659+0000", 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419496+0000", 2024-04-20T02:52:52.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00083715799999999996 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419513+0000", 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7489000000000001e-05 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419524+0000", 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0118e-05 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429881+0000", 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010357762 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429905+0000", 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3546000000000001e-05 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468790+0000", 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038884821 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468832+0000", 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2341e-05 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3898 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'133, mlcod=133'129)", 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 7.9951435059999998, 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038834664999999997, 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.538 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470817+0000", 2024-04-20T02:52:52.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471051+0000", 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023384600000000001 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471066+0000", 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4205e-05 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471074+0000", 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4090000000000003e-06 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472816+0000", 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017419429999999999 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472843+0000", 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7296000000000001e-05 2024-04-20T02:52:52.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509592+0000", 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036748496999999998 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509653+0000", 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1707999999999999e-05 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3899 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'134, mlcod=133'129)", 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 7.9778770699999999, 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038218993999999999, 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488084+0000", 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488262+0000", 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017768600000000001 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488274+0000", 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2388000000000001e-05 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488283+0000", 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3840000000000003e-06 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489094+0000", 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081045100000000001 2024-04-20T02:52:52.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489119+0000", 2024-04-20T02:52:52.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4751000000000001e-05 2024-04-20T02:52:52.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526264+0000", 2024-04-20T02:52:52.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037144912000000002 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526304+0000", 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0513000000000003e-05 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3891 4.7 e146/145 4:e2c0d65b:::smithi10548823-46:head v 146'105, mlcod=143'100)", 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 7.9768969749999998, 2024-04-20T02:52:52.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041132308999999999, 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489072+0000", 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9539999999999998e-06 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489082+0000", 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0329000000000001e-05 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489089+0000", 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4959999999999999e-06 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489814+0000", 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072423699999999997 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489837+0000", 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3218999999999999e-05 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530157+0000", 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040319762000000002 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530198+0000", 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0886000000000001e-05 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.551 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.7 146/145 [PushOp(4:e2c0d65b:::smithi10548823-46:43, version: 143'96, data_included: [426944~820638,1468778~227938], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:e2c0d65b:::smithi10548823-46:43@143'96, size: 3311012, copy_subset: [0~3311012], clone_subset: {}, snapset: 45=[]:{3f=[3f,39],43=[41],45=[45]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1696716, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 7.9756856230000004, 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039876652999999998, 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:52:52.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490276+0000", 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490785+0000", 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050941300000000001 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490804+0000", 2024-04-20T02:52:52.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9080999999999999e-05 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490823+0000", 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.855e-05 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498138+0000", 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007315004 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498153+0000", 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5515e-05 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530153+0000", 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031999845999999998 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4016 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'146, mlcod=147'143)", 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.2033813770000004, 2024-04-20T02:52:52.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054600045999999999, 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262580+0000", 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262733+0000", 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015289700000000001 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262746+0000", 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.375e-05 2024-04-20T02:52:52.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262753+0000", 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6159999999999998e-06 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262817+0000", 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4202999999999995e-05 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262842+0000", 2024-04-20T02:52:52.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5296e-05 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317137+0000", 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054294664999999999 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317181+0000", 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4017 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'147, mlcod=147'143)", 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.1883346909999997, 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039585437000000001, 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277626+0000", 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278058+0000", 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043228299999999999 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278074+0000", 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5775000000000001e-05 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278083+0000", 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2620000000000004e-06 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286849+0000", 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087652210000000001 2024-04-20T02:52:52.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286881+0000", 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1919999999999999e-05 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317188+0000", 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030307392999999998 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317213+0000", 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4953999999999999e-05 2024-04-20T02:52:52.563 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.564 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.564 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.564 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:52.564 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:52.564 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:52.565 INFO:tasks.rados.rados.0.smithi105.stdout:1769: expect (ObjNum 294 snap 32 seq_num 294) 2024-04-20T02:52:52.565 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:52.566+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:52.566 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:52.567 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 2.6 --file /var/log/ceph/exp.2.6.4 2024-04-20T02:52:52.666 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:52.667+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:52.739 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:52.740 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:52.740 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:52:52.740 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:52.750 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:52.752 INFO:tasks.rados.rados.0.smithi105.stdout:1782: expect (ObjNum 317 snap 34 seq_num 317) 2024-04-20T02:52:52.793 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:52.794+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:52.867 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:52.868+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:52.878 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:52.879 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:52.879 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:52.879 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:52.879 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:52.888 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:52.940 INFO:tasks.rados.rados.0.smithi105.stdout:1755: expect (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:52:52.972 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:52.973+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:53.069 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 103.619016628, 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.070 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:53.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:52:53.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:52:53.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.568833861000002, 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:52:53.074 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:52:53.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:53.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.568768411999997, 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 94.568742913999998, 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:53.083 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:52:53.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:53.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:52:53.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2524 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1723287~549075] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 64.781233431000004, 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054106970999999997, 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.088 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2524 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286419+0000", 2024-04-20T02:52:53.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286924+0000", 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050494600000000004 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286944+0000", 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9833000000000001e-05 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286957+0000", 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2901000000000001e-05 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298074+0000", 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011117320999999999 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298151+0000", 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6909999999999994e-05 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:53.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298274+0000", 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012279700000000001 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326267+0000", 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027993284 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331226+0000", 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049590559999999999 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340442+0000", 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092155740000000007 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340488+0000", 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6208000000000002e-05 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340528+0000", 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0277000000000001e-05 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2525 4.8 4:1afa5d2b:::smithi10548823-17:head [write 2781837~703343] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 64.774482216999999, 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050020689, 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2525 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293173+0000", 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293172+0000", 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293838+0000", 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00066662700000000004 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293857+0000", 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.929e-05 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.293867+0000", 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9369999999999999e-06 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310149+0000", 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016281713999999999 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310257+0000", 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108262 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.310439+0000", 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018146300000000001 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.327814+0000", 2024-04-20T02:52:53.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017375043 2024-04-20T02:52:53.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331934+0000", 2024-04-20T02:52:53.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041200489999999998 2024-04-20T02:52:53.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343112+0000", 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011177991999999999 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343146+0000", 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3813000000000003e-05 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.343193+0000", 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7469999999999998e-05 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.099 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3637 4.1 e137/60 4:80c7a726:::smithi10548823-19:head v 137'124, mlcod=120'120)", 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 17.443245871999999, 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052640605, 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624408+0000", 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624537+0000", 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012880199999999999 2024-04-20T02:52:53.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624548+0000", 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0216e-05 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624556+0000", 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6179999999999995e-06 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624603+0000", 2024-04-20T02:52:53.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6601999999999999e-05 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624632+0000", 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9277000000000001e-05 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.676999+0000", 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052366639999999999 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.677049+0000", 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0822999999999998e-05 2024-04-20T02:52:53.103 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3880 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'231, mlcod=141'228)", 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6607508919999994, 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.061291667000000001, 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:52:53.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406903+0000", 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407359+0000", 2024-04-20T02:52:53.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00045623700000000002 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407370+0000", 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1052e-05 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407377+0000", 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1080000000000004e-06 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407429+0000", 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2033999999999997e-05 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407458+0000", 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8050000000000001e-05 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468161+0000", 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060703628000000003 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468196+0000", 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4344999999999997e-05 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3888 4.7 4:e2c0d65b:::smithi10548823-46:head [write 413538~536345,stat] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6421977640000005, 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.059175945000000001, 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.108 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3888 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425456+0000", 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426097+0000", 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00064115099999999996 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426113+0000", 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6118000000000001e-05 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426122+0000", 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.115e-06 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426179+0000", 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6815999999999999e-05 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426316+0000", 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000136606 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426499+0000", 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000183191 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443652+0000", 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017153471999999999 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484036+0000", 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040384097000000001 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484231+0000", 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000194811 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484363+0000", 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013146300000000001 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484426+0000", 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3563000000000001e-05 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484633+0000", 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020662299999999999 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3889 4.7 4:e2c0d65b:::smithi10548823-46:head [write 1395221~583781] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.635303789, 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.057918795000000002, 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.114 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3889 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432350+0000", 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433490+0000", 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001140606 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433506+0000", 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5828000000000002e-05 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433515+0000", 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8279999999999992e-06 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443692+0000", 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010177222 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443839+0000", 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014662 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.444072+0000", 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023362200000000001 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484114+0000", 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040041641000000003 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484244+0000", 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012987200000000001 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484279+0000", 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5301e-05 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490055+0000", 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057763459999999999 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490115+0000", 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9320999999999997e-05 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490270+0000", 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015498199999999999 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3890 4.7 4:e2c0d65b:::smithi10548823-46:head [write 2556103~658617] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6272896879999994, 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066878325000000002, 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.120 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3890 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440363+0000", 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441831+0000", 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001467632 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441842+0000", 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0540000000000001e-05 2024-04-20T02:52:53.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441850+0000", 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7260000000000008e-06 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463536+0000", 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02168544 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463681+0000", 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000145086 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463941+0000", 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00025968600000000002 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484317+0000", 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020376452 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490318+0000", 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060012490000000002 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490398+0000", 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9598000000000006e-05 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506988+0000", 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016589706999999999 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507066+0000", 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8299000000000004e-05 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507243+0000", 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000177503 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3891 4.7 4:e2c0d65b:::smithi10548823-46:head [setxattr _header (57) in=7b,truncate 3613538] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6257941789999997, 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088590031, 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.126 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3891 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:52:53.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441859+0000", 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441864+0000", 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8069999999999999e-06 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441869+0000", 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0989999999999998e-06 2024-04-20T02:52:53.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441875+0000", 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5820000000000001e-06 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484675+0000", 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042799867999999998 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484817+0000", 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014236899999999999 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485048+0000", 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023057199999999999 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490359+0000", 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053116819999999999 2024-04-20T02:52:53.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506963+0000", 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016603562999999998 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507256+0000", 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00029288699999999998 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530316+0000", 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023060343000000001 2024-04-20T02:52:53.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530356+0000", 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9301999999999997e-05 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530451+0000", 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5037999999999994e-05 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.132 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3892 4.7 4:e2c0d65b:::smithi10548823-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6257744909999996, 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088912482000000001, 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3892 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:53.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441883+0000", 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9579999999999999e-06 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441889+0000", 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7499999999999997e-06 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441895+0000", 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485431+0000", 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043536321000000003 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485537+0000", 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010612599999999999 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530489+0000", 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044951996000000001 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530598+0000", 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108937 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530793+0000", 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019441 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3898 4.8 4:1c68e542:::smithi10548823-8:head [write 447152~403196] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:52:53.137 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.6014314079999998, 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.056917473000000003, 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3898 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466222+0000", 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466722+0000", 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00049941299999999998 2024-04-20T02:52:53.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466734+0000", 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2860999999999999e-05 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466743+0000", 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7159999999999992e-06 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466809+0000", 2024-04-20T02:52:53.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6165000000000001e-05 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466928+0000", 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011869199999999999 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.467158+0000", 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000230251 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498338+0000", 2024-04-20T02:52:53.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.03117996 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.522958+0000", 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024619318000000001 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523002+0000", 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4675999999999997e-05 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523082+0000", 2024-04-20T02:52:53.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9803999999999997e-05 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523141+0000", 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8767000000000002e-05 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3899 4.8 4:1c68e542:::smithi10548823-8:head [write 1512185~587450] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.5935036409999999, 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060518556000000001, 2024-04-20T02:52:53.143 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3899 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474149+0000", 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474686+0000", 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053696199999999999 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474703+0000", 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.615e-05 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474713+0000", 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0829000000000001e-05 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478533+0000", 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003819687 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478640+0000", 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106974 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478766+0000", 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125859 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523803+0000", 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045036616000000002 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526432+0000", 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026291489999999999 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534589+0000", 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081572209999999992 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534623+0000", 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534670+0000", 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6968999999999997e-05 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4016 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 665643~697786] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.8063348760000002, 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060676251, 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.149 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4016 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:52:53.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261318+0000", 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262081+0000", 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076285299999999997 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262095+0000", 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3986e-05 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262118+0000", 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2407000000000001e-05 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262164+0000", 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6248999999999997e-05 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262238+0000", 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4202000000000005e-05 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262363+0000", 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012437799999999999 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314944+0000", 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052581687000000002 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317407+0000", 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024628300000000001 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321887+0000", 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0044803020000000002 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321941+0000", 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3769e-05 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321996+0000", 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4988000000000003e-05 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4017 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 2039745~753122] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.7987050780000002, 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053146383999999998, 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.155 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4017 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268948+0000", 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269678+0000", 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072975799999999995 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269693+0000", 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.453e-05 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269704+0000", 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1623e-05 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277190+0000", 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074859849999999997 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277276+0000", 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5809999999999993e-05 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277405+0000", 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012914600000000001 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314952+0000", 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037546931999999998 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317441+0000", 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024889840000000001 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322031+0000", 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004589465 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322059+0000", 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322096+0000", 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591999999999998e-05 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.160 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4074 4.11 4:8b91c759:::smithi10548823-5:head [write 1591248~408851] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.92573768899999997, 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053876146, 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4074 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.161 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141916+0000", 2024-04-20T02:52:53.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142740+0000", 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000823616 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142750+0000", 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6150000000000003e-06 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142756+0000", 2024-04-20T02:52:53.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9419999999999998e-06 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2061999999999997e-05 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142815+0000", 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5050000000000004e-06 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174069+0000", 2024-04-20T02:52:53.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031253933999999997 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174193+0000", 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012386 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174438+0000", 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00024561400000000001 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187472+0000", 2024-04-20T02:52:53.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013033375999999999 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.188278+0000", 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00080595500000000002 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195551+0000", 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072733340000000002 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195631+0000", 2024-04-20T02:52:53.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9882999999999994e-05 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195793+0000", 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000162223 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4075 4.11 4:8b91c759:::smithi10548823-5:head [setxattr _header (56) in=7b,truncate 2000099] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.924886757, 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084064655000000002, 2024-04-20T02:52:53.167 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4075 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:52:53.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142767+0000", 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142778+0000", 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0709e-05 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142783+0000", 2024-04-20T02:52:53.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4539999999999998e-06 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142788+0000", 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2850000000000002e-06 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142845+0000", 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6919000000000002e-05 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142856+0000", 2024-04-20T02:52:53.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0936e-05 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186635+0000", 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043778897999999997 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186756+0000", 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121087 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186967+0000", 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021064599999999999 2024-04-20T02:52:53.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.189477+0000", 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025104929999999999 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.190395+0000", 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00091717399999999998 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226595+0000", 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036200195999999997 2024-04-20T02:52:53.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226661+0000", 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6465999999999997e-05 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226833+0000", 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000171471 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.173 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4076 4.11 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 0.92484575099999999, 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084363853000000003, 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4076 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142808+0000", 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142811+0000", 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9229999999999998e-06 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142818+0000", 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4420000000000003e-06 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142823+0000", 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5199999999999997e-06 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.176 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142875+0000", 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1759999999999999e-05 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142880+0000", 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6639999999999997e-06 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187624+0000", 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044743949999999998 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187726+0000", 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0001022 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226882+0000", 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039155862 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227003+0000", 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120715 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:53.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:53.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227173+0000", 2024-04-20T02:52:53.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017032899999999999 2024-04-20T02:52:53.179 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.179 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.179 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.179 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:53.179 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:53.179 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:53.180 INFO:tasks.rados.rados.0.smithi105.stdout:1763: finishing write tid 1 to smithi10548823-5 2024-04-20T02:52:53.180 INFO:tasks.rados.rados.0.smithi105.stdout:1763: finishing write tid 2 to smithi10548823-5 2024-04-20T02:52:53.180 INFO:tasks.rados.rados.0.smithi105.stdout:1752: expect (ObjNum 342 snap 37 seq_num 342) 2024-04-20T02:52:53.181 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:53.181+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:53.181 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:53.282 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:53.282+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:53.285 INFO:tasks.rados.rados.0.smithi105.stdout:1779: got (2) No such file or directory 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1779: got expected ENOENT (src dne) 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1761: finishing write tid 1 to smithi10548823-44 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1761: finishing write tid 2 to smithi10548823-44 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1752: done (15 left) 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1754: done (14 left) 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1755: done (13 left) 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1769: done (12 left) 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1779: done (11 left) 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1782: done (10 left) 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1787: done (9 left) 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1792: read oid 50 snap 35 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1792: expect (ObjNum 285 snap 29 seq_num 285) 2024-04-20T02:52:53.286 INFO:tasks.rados.rados.0.smithi105.stdout:1793: read oid 22 snap -1 2024-04-20T02:52:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1793: expect (ObjNum 361 snap 37 seq_num 361) 2024-04-20T02:52:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1794: write (excl) oid 21 current snap is 37 2024-04-20T02:52:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1759: finishing write tid 1 to smithi10548823-6 2024-04-20T02:52:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1759: finishing write tid 2 to smithi10548823-6 2024-04-20T02:52:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1759: finishing write tid 3 to smithi10548823-6 2024-04-20T02:52:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1760: finishing write tid 1 to smithi10548823-24 2024-04-20T02:52:53.287 INFO:tasks.rados.rados.0.smithi105.stdout:1794: seq_num 365 ranges {534555=650829,1659104=475451} 2024-04-20T02:52:53.291 INFO:tasks.rados.rados.0.smithi105.stdout:1794: writing smithi10548823-21 from 534555 to 1185384 tid 1 2024-04-20T02:52:53.298 INFO:tasks.rados.rados.0.smithi105.stdout:1794: writing smithi10548823-21 from 1659104 to 2134555 tid 2 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1795: read oid 3 snap -1 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1795: expect deleted 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1796: write (excl) oid 31 current snap is 37 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1760: finishing write tid 2 to smithi10548823-24 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1763: finishing write tid 3 to smithi10548823-5 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1763: finishing write tid 4 to smithi10548823-5 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1763: oid 5 updating version 0 to 209 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1763: oid 5 version 209 is already newer than 207 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1763: oid 5 version 209 is already newer than 208 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 209 (ObjNum 357 snap 37 seq_num 357) dirty exists 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1763: left oid 5 (ObjNum 357 snap 37 seq_num 357) 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1759: finishing write tid 4 to smithi10548823-6 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1759: finishing write tid 5 to smithi10548823-6 2024-04-20T02:52:53.299 INFO:tasks.rados.rados.0.smithi105.stdout:1759: oid 6 updating version 0 to 277 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1759: oid 6 version 277 is already newer than 276 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1759: oid 6 updating version 277 to 279 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1759: oid 6 version 279 is already newer than 278 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 279 (ObjNum 354 snap 37 seq_num 354) dirty exists 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1759: left oid 6 (ObjNum 354 snap 37 seq_num 354) 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1789: finishing set_chunk to oid 20 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 20 set_chunk returned error code -95 offset: 1438720 length: 1129472 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 0 (ObjNum 326 snap 35 seq_num 326) dirty exists 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1761: finishing write tid 3 to smithi10548823-44 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1761: finishing write tid 4 to smithi10548823-44 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1761: finishing write tid 5 to smithi10548823-44 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1761: oid 44 updating version 0 to 128 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1761: oid 44 updating version 128 to 131 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1761: oid 44 version 131 is already newer than 129 2024-04-20T02:52:53.300 INFO:tasks.rados.rados.0.smithi105.stdout:1761: oid 44 version 131 is already newer than 130 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 131 (ObjNum 356 snap 37 seq_num 356) dirty exists 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1761: left oid 44 (ObjNum 356 snap 37 seq_num 356) 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1773: finishing write tid 1 to smithi10548823-18 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1773: finishing write tid 2 to smithi10548823-18 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1760: finishing write tid 3 to smithi10548823-24 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1760: finishing write tid 4 to smithi10548823-24 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1760: finishing write tid 5 to smithi10548823-24 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1760: oid 24 updating version 0 to 433 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1760: oid 24 updating version 433 to 435 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1760: oid 24 updating version 435 to 436 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1760: oid 24 version 436 is already newer than 434 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 436 (ObjNum 355 snap 37 seq_num 355) dirty exists 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1760: left oid 24 (ObjNum 355 snap 37 seq_num 355) 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1770: finishing write tid 1 to smithi10548823-36 2024-04-20T02:52:53.301 INFO:tasks.rados.rados.0.smithi105.stdout:1791: expect (ObjNum 325 snap 35 seq_num 325) 2024-04-20T02:52:53.328 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:53.328 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:53.328 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:53.328 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:53.340 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:53.383 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:53.384+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:53.470 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:53.470 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:53.471 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:53.471 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:53.471 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:53.481 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:53.484 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:53.485+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:53.517 INFO:tasks.rados.rados.0.smithi105.stdout:1770: finishing write tid 2 to smithi10548823-36 2024-04-20T02:52:53.517 INFO:tasks.rados.rados.0.smithi105.stdout:1796: seq_num 366 ranges {551362=726078,1717586=409779} 2024-04-20T02:52:53.524 INFO:tasks.rados.rados.0.smithi105.stdout:1796: writing smithi10548823-31 from 551362 to 1277440 tid 1 2024-04-20T02:52:53.529 INFO:tasks.rados.rados.0.smithi105.stdout:1796: writing smithi10548823-31 from 1717586 to 2127365 tid 2 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1759: done (13 left) 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1760: done (12 left) 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1761: done (11 left) 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1763: done (10 left) 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1789: done (9 left) 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1791: done (8 left) 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1797: write oid 27 current snap is 37 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1770: finishing write tid 3 to smithi10548823-36 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1770: finishing write tid 4 to smithi10548823-36 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1770: oid 36 updating version 0 to 438 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1770: oid 36 version 438 is already newer than 437 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:1770: oid 36 updating version 438 to 439 2024-04-20T02:52:53.530 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 439 (ObjNum 360 snap 37 seq_num 360) dirty exists 2024-04-20T02:52:53.531 INFO:tasks.rados.rados.0.smithi105.stdout:1770: left oid 36 (ObjNum 360 snap 37 seq_num 360) 2024-04-20T02:52:53.531 INFO:tasks.rados.rados.0.smithi105.stdout:1773: finishing write tid 3 to smithi10548823-18 2024-04-20T02:52:53.531 INFO:tasks.rados.rados.0.smithi105.stdout:1773: finishing write tid 4 to smithi10548823-18 2024-04-20T02:52:53.531 INFO:tasks.rados.rados.0.smithi105.stdout:1773: oid 18 updating version 0 to 135 2024-04-20T02:52:53.531 INFO:tasks.rados.rados.0.smithi105.stdout:1773: oid 18 version 135 is already newer than 133 2024-04-20T02:52:53.531 INFO:tasks.rados.rados.0.smithi105.stdout:1773: oid 18 version 135 is already newer than 134 2024-04-20T02:52:53.531 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 135 (ObjNum 362 snap 37 seq_num 362) dirty exists 2024-04-20T02:52:53.531 INFO:tasks.rados.rados.0.smithi105.stdout:1773: left oid 18 (ObjNum 362 snap 37 seq_num 362) 2024-04-20T02:52:53.531 INFO:tasks.rados.rados.0.smithi105.stdout:1790: finishing write tid 1 to smithi10548823-29 2024-04-20T02:52:53.531 INFO:tasks.rados.rados.0.smithi105.stdout:1797: seq_num 367 ranges {568169=401327,1376068=669790} 2024-04-20T02:52:53.535 INFO:tasks.rados.rados.0.smithi105.stdout:1797: writing smithi10548823-27 from 568169 to 969496 tid 1 2024-04-20T02:52:53.542 INFO:tasks.rados.rados.0.smithi105.stdout:1797: writing smithi10548823-27 from 1376068 to 2045858 tid 2 2024-04-20T02:52:53.542 INFO:tasks.rados.rados.0.smithi105.stdout:1770: done (8 left) 2024-04-20T02:52:53.542 INFO:tasks.rados.rados.0.smithi105.stdout:1773: done (7 left) 2024-04-20T02:52:53.542 INFO:tasks.rados.rados.0.smithi105.stdout:1798: set_chunk oid 47 target oid 2024-04-20T02:52:53.585 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:53.586+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:53.607 INFO:tasks.rados.rados.0.smithi105.stdout:1798: set_chunk oid 47 offset: 116736 length: 3247104 target oid 38e41cf72267810287cbccdcc608985df29398c5b7dfa90f8aeb3e7bc4dba1db offset: 0 snap -1 2024-04-20T02:52:53.607 INFO:tasks.rados.rados.0.smithi105.stdout:1799: set_chunk oid 33 target oid 2024-04-20T02:52:53.631 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:53.631 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:53.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:53.631 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-04-20T02:52:53.632 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:53.639 INFO:tasks.rados.rados.0.smithi105.stdout:1799: set_chunk oid 33 offset: 1062400 length: 1223168 target oid 1122d562691b4b390a06d30f9996a3cf9717656fd97da73fefeec349aaded992 offset: 0 snap -1 2024-04-20T02:52:53.639 INFO:tasks.rados.rados.0.smithi105.stdout:1800: set_chunk oid 46 target oid 2024-04-20T02:52:53.642 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:53.705 INFO:teuthology.orchestra.run.smithi138.stderr:Exporting 2.6 info 2.6( empty local-lis/les=18/19 n=0 ec=18/18 lis/c=18/18 les/c/f=19/19/0 sis=18) 2024-04-20T02:52:53.705 INFO:teuthology.orchestra.run.smithi138.stderr:Export successful 2024-04-20T02:52:53.715 INFO:tasks.rados.rados.0.smithi105.stdout:1800: set_chunk oid 46 offset: 944640 length: 2392064 target oid da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb offset: 0 snap -1 2024-04-20T02:52:53.715 INFO:tasks.rados.rados.0.smithi105.stdout:1801: read oid 48 snap 32 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:1801: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:1802: write oid 5 current snap is 37 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:1790: finishing write tid 2 to smithi10548823-29 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:1790: finishing write tid 3 to smithi10548823-29 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:1790: finishing write tid 4 to smithi10548823-29 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:1790: oid 29 updating version 0 to 38 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:1790: oid 29 updating version 38 to 39 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:1790: oid 29 updating version 39 to 40 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 40 (ObjNum 364 snap 37 seq_num 364) dirty exists 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:1790: left oid 29 (ObjNum 364 snap 37 seq_num 364) 2024-04-20T02:52:53.716 INFO:tasks.rados.rados.0.smithi105.stdout:1802: seq_num 368 ranges {584976=476576,1834550=350426} 2024-04-20T02:52:53.716 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:53.716+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:53.721 INFO:tasks.rados.rados.0.smithi105.stdout:1802: writing smithi10548823-5 from 584976 to 1061552 tid 1 2024-04-20T02:52:53.727 INFO:tasks.rados.rados.0.smithi105.stdout:1802: writing smithi10548823-5 from 1834550 to 2184976 tid 2 2024-04-20T02:52:53.727 INFO:tasks.rados.rados.0.smithi105.stdout:1790: done (11 left) 2024-04-20T02:52:53.727 INFO:tasks.rados.rados.0.smithi105.stdout:1803: set_chunk oid 41 target oid 2024-04-20T02:52:53.768 INFO:tasks.rados.rados.0.smithi105.stdout:1803: set_chunk oid 41 offset: 171008 length: 2797056 target oid c48645e094c5e61dd2660a0b6d13eff065486466cd289a028825bf2820823db6 offset: 0 snap -1 2024-04-20T02:52:53.768 INFO:tasks.rados.rados.0.smithi105.stdout:1804: read oid 49 snap -1 2024-04-20T02:52:53.768 INFO:tasks.rados.rados.0.smithi105.stdout:1804: expect (ObjNum 352 snap 37 seq_num 352) 2024-04-20T02:52:53.769 INFO:tasks.rados.rados.0.smithi105.stdout:1793: expect (ObjNum 361 snap 37 seq_num 361) 2024-04-20T02:52:53.786 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:53.787+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:53.819 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:53.819 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:53.819 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:53.819 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:53.829 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:53.887 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:53.887+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:53.937 INFO:tasks.rados.rados.0.smithi105.stdout:1792: expect (ObjNum 285 snap 29 seq_num 285) 2024-04-20T02:52:53.988 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:53.988+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:53.998 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:53.998 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:53.998 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:53.998 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:53.998 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:54.009 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:54.087 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:54.088 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:54.089+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:54.132 INFO:tasks.rados.rados.0.smithi105.stdout:1794: finishing write tid 1 to smithi10548823-21 2024-04-20T02:52:54.132 INFO:tasks.rados.rados.0.smithi105.stdout:1794: finishing write tid 2 to smithi10548823-21 2024-04-20T02:52:54.132 INFO:tasks.rados.rados.0.smithi105.stdout:1794: finishing write tid 3 to smithi10548823-21 2024-04-20T02:52:54.132 INFO:tasks.rados.rados.0.smithi105.stdout:1794: finishing write tid 4 to smithi10548823-21 2024-04-20T02:52:54.132 INFO:tasks.rados.rados.0.smithi105.stdout:1794: oid 21 updating version 0 to 170 2024-04-20T02:52:54.132 INFO:tasks.rados.rados.0.smithi105.stdout:1794: oid 21 updating version 170 to 171 2024-04-20T02:52:54.132 INFO:tasks.rados.rados.0.smithi105.stdout:1794: oid 21 updating version 171 to 172 2024-04-20T02:52:54.132 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 172 (ObjNum 365 snap 37 seq_num 365) dirty exists 2024-04-20T02:52:54.132 INFO:tasks.rados.rados.0.smithi105.stdout:1794: left oid 21 (ObjNum 365 snap 37 seq_num 365) 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1796: finishing write tid 1 to smithi10548823-31 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1796: finishing write tid 2 to smithi10548823-31 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1796: finishing write tid 3 to smithi10548823-31 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1796: finishing write tid 4 to smithi10548823-31 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1796: oid 31 updating version 0 to 252 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1796: oid 31 version 252 is already newer than 251 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1796: oid 31 updating version 252 to 253 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 253 (ObjNum 366 snap 37 seq_num 366) dirty exists 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1796: left oid 31 (ObjNum 366 snap 37 seq_num 366) 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1797: finishing write tid 1 to smithi10548823-27 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1797: finishing write tid 2 to smithi10548823-27 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1797: finishing write tid 3 to smithi10548823-27 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1797: finishing write tid 4 to smithi10548823-27 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1797: oid 27 updating version 0 to 283 2024-04-20T02:52:54.133 INFO:tasks.rados.rados.0.smithi105.stdout:1797: oid 27 version 283 is already newer than 282 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:1797: oid 27 version 283 is already newer than 281 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 283 (ObjNum 367 snap 37 seq_num 367) dirty exists 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:1797: left oid 27 (ObjNum 367 snap 37 seq_num 367) 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:1798: finishing set_chunk to oid 47 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 149 (ObjNum 349 snap 37 seq_num 349) dirty exists 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:1799: finishing set_chunk to oid 33 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 33 set_chunk returned error code -95 offset: 1062400 length: 1223168 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 0 (ObjNum 196 snap 19 seq_num 196) dirty exists 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:1805: read oid 2 snap 32 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:1805: expect (ObjNum 302 snap 32 seq_num 302) 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:1792: done (13 left) 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:1793: done (12 left) 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:1794: done (11 left) 2024-04-20T02:52:54.134 INFO:tasks.rados.rados.0.smithi105.stdout:1795: done (10 left) 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1796: done (9 left) 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1797: done (8 left) 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1798: done (7 left) 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1799: done (6 left) 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1806: read oid 30 snap 35 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1806: expect (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1807: read oid 25 snap 32 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1807: expect (ObjNum 294 snap 32 seq_num 294) 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1808: rollback oid 44 current snap is 37 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 44 to 32 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1800: finishing set_chunk to oid 46 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 105 (ObjNum 334 snap 36 seq_num 334) dirty exists 2024-04-20T02:52:54.135 INFO:tasks.rados.rados.0.smithi105.stdout:1801: expect (ObjNum 195 snap 19 seq_num 195) 2024-04-20T02:52:54.150 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:54.150 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 127.603818272, 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:52:54.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:52:54.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:52:54.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 123.215895811, 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.155 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 123.215635605, 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.161 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:52:54.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:52:54.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:52:54.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:52:54.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.168 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 123.215614137, 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:52:54.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 114.021942205, 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:52:54.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:52:54.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:52:54.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.179 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2292 4.3 4:c07c6daa:::smithi10548823-7:head [write 544593~799729] snapc 1e=[1e,1d,1b,18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 79.988899012999994, 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061161747000000002, 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2292 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159577+0000", 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160019+0000", 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000442674 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160045+0000", 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5446e-05 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160058+0000", 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3444e-05 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160487+0000", 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000428899 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160616+0000", 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012861099999999999 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160838+0000", 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000222415 2024-04-20T02:52:54.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207115+0000", 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046276543000000003 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207164+0000", 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9497999999999999e-05 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220657+0000", 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013492386 2024-04-20T02:52:54.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220698+0000", 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1451e-05 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220741+0000", 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3078000000000003e-05 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.185 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2699 4.0 e95/60 4:007b2a87:::smithi10548823-39:head v 95'100, mlcod=94'97)", 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 61.799317725000002, 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061625529999999998, 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349160+0000", 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349351+0000", 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019061600000000001 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349364+0000", 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2835e-05 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349376+0000", 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349466+0000", 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9615000000000002e-05 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349492+0000", 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6494000000000001e-05 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.410234+0000", 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060741334000000001 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.410786+0000", 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000552539 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.189 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3423 4.3 4:c07c6daa:::smithi10548823-7:head [write 558065~619869] snapc 3a=[3a,39,38,36,35,33] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 25.759029989999998, 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061409060000000001, 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3423 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.190 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389446+0000", 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:54.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389586+0000", 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014017899999999999 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389605+0000", 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389617+0000", 2024-04-20T02:52:54.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1956999999999999e-05 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.412832+0000", 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023215612 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.412909+0000", 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6555000000000001e-05 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:54.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.413035+0000", 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000126599 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450700+0000", 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037664717 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450712+0000", 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1422e-05 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450741+0000", 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9791999999999999e-05 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450801+0000", 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9199e-05 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450857+0000", 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6882999999999997e-05 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.195 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3771 4.b 4:d4308fef:::smithi10548823-6:head [write 2548663~763022] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.966585479999999, 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072733780999999997, 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3771 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.196 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181891+0000", 2024-04-20T02:52:54.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182301+0000", 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040985099999999998 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182317+0000", 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6305999999999999e-05 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182329+0000", 2024-04-20T02:52:54.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1337e-05 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198720+0000", 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016391606 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198807+0000", 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7319999999999997e-05 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198925+0000", 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117345 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218062+0000", 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019137134 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234990+0000", 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016928367 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254488+0000", 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019497460000000001 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254571+0000", 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3029999999999996e-05 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254627+0000", 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5865000000000001e-05 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.201 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3772 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3395047] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.966138908, 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072419708999999999, 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3772 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182343+0000", 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0099999999999997e-06 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182349+0000", 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9569999999999999e-06 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182354+0000", 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.826e-06 2024-04-20T02:52:54.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217228+0000", 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034874555000000002 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217312+0000", 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3478000000000003e-05 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:54.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217431+0000", 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119512 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234969+0000", 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017537833999999999 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235012+0000", 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3411000000000002e-05 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254638+0000", 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019625815000000001 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254682+0000", 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3945e-05 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254759+0000", 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6882e-05 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.207 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3773 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.966120301, 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072538680999999994, 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3773 2024-04-20T02:52:54.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:52:54.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182360+0000", 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3309999999999998e-06 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182366+0000", 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.981e-06 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182371+0000", 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5390000000000001e-06 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217932+0000", 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035561700000000002 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218020+0000", 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7352e-05 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254778+0000", 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036757971 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254819+0000", 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1525e-05 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254897+0000", 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7621999999999996e-05 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.212 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3774 4.3 4:c07c6daa:::smithi10548823-7:head [write 611854~717004] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.190321+0000", 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.958157148, 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064562913999999999, 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3774 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190321+0000", 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190321+0000", 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190319+0000", 2024-04-20T02:52:54.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190704+0000", 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038458399999999999 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190721+0000", 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6988000000000001e-05 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190734+0000", 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.276e-05 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217683+0000", 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026949832 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217775+0000", 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1669000000000006e-05 2024-04-20T02:52:54.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217991+0000", 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021635400000000001 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254534+0000", 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036542842999999998 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254573+0000", 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8921000000000002e-05 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254780+0000", 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020691700000000001 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254833+0000", 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2785999999999999e-05 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254884+0000", 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1159999999999998e-05 2024-04-20T02:52:54.218 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3775 4.3 4:c07c6daa:::smithi10548823-7:head [write 2126143~561767] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.198678+0000", 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.949799887999999, 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062905904999999998, 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.219 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3775 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198678+0000", 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198678+0000", 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198677+0000", 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198991+0000", 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031458700000000002 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.199005+0000", 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3134e-05 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.199013+0000", 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3489999999999992e-06 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234979+0000", 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035965991000000003 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235102+0000", 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012322400000000001 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235329+0000", 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022682799999999999 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254591+0000", 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019262029999999999 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254607+0000", 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6150999999999998e-05 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.261494+0000", 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068870779999999996 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.261539+0000", 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4400000000000002e-05 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.261584+0000", 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5775000000000001e-05 2024-04-20T02:52:54.224 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3880 4.3 4:c32272c0:::smithi10548823-41:head [write 796731~461096] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.406265+0000", 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.7422133829999993, 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066110050000000004, 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.225 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3880 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406265+0000", 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406265+0000", 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406264+0000", 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406481+0000", 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021666200000000001 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406493+0000", 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406499+0000", 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3609999999999997e-06 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406540+0000", 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0728e-05 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406632+0000", 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1821000000000001e-05 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406742+0000", 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110178 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.459209+0000", 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.052467645 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.459342+0000", 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013213 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.472265+0000", 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012923128000000001 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.472321+0000", 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.63e-05 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.472375+0000", 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4042999999999997e-05 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.230 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4057 4.b 4:d4308fef:::smithi10548823-6:head [write 749678~790384,stat] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.0052655609999999, 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074563737000000005, 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4057 2024-04-20T02:52:54.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143210+0000", 2024-04-20T02:52:54.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143760+0000", 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055015999999999995 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143772+0000", 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1345000000000001e-05 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143781+0000", 2024-04-20T02:52:54.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0929999999999992e-06 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143895+0000", 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113998 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143906+0000", 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1195999999999999e-05 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.153590+0000", 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096836000000000005 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155611+0000", 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002021012 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217691+0000", 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062080244 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217777+0000", 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5583999999999999e-05 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4058 4.b 4:d4308fef:::smithi10548823-6:head [write 2332155~504118] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.004684535, 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074090323999999999, 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:54.236 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4058 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:52:54.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143793+0000", 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144104+0000", 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000310971 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144112+0000", 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1990000000000006e-06 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144118+0000", 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.443e-06 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144162+0000", 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3066999999999999e-05 2024-04-20T02:52:54.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144172+0000", 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0083e-05 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155911+0000", 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011739657000000001 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:52:54.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155993+0000", 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1754000000000004e-05 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217810+0000", 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061816958999999998 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217884+0000", 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4098999999999996e-05 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4059 4.b 4:d4308fef:::smithi10548823-6:head [write 3271857~549469] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.0043483740000001, 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073860173000000001, 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4059 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144129+0000", 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144439+0000", 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030924000000000001 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144447+0000", 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8930000000000005e-06 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144455+0000", 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1410000000000005e-06 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144505+0000", 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0698000000000001e-05 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144520+0000", 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4605e-05 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156034+0000", 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011513809 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156130+0000", 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6039999999999995e-05 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217915+0000", 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061785501999999999 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217990+0000", 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4770000000000004e-05 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4060 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3949678] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.0040129950000001, 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085188106999999999, 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4060 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144469+0000", 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.946e-06 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144476+0000", 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7909999999999999e-06 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144482+0000", 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.601e-06 2024-04-20T02:52:54.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144803+0000", 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032103699999999998 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144813+0000", 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0183999999999999e-05 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156168+0000", 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011354523 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156254+0000", 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5706999999999998e-05 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218023+0000", 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061769566999999997 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218137+0000", 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113912 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218377+0000", 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023941000000000001 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225107+0000", 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067304510000000001 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225121+0000", 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4341e-05 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229481+0000", 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043599880000000004 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229522+0000", 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0224e-05 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229653+0000", 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013190500000000001 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4061 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 2.0038733780000002, 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085431572999999997, 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.254 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4061 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:52:54.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144604+0000", 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144608+0000", 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1080000000000003e-06 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144618+0000", 2024-04-20T02:52:54.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5149999999999995e-06 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144625+0000", 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8260000000000004e-06 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144846+0000", 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022160699999999999 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144854+0000", 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0399999999999993e-06 2024-04-20T02:52:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156327+0000", 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011472178 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156457+0000", 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013012599999999999 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218704+0000", 2024-04-20T02:52:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062247080000000003 2024-04-20T02:52:54.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:54.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218795+0000", 2024-04-20T02:52:54.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1391000000000006e-05 2024-04-20T02:52:54.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.259 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:52:54.262 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 2.6 2024-04-20T02:52:54.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229703+0000", 2024-04-20T02:52:54.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010907834999999999 2024-04-20T02:52:54.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229818+0000", 2024-04-20T02:52:54.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115423 2024-04-20T02:52:54.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.230037+0000", 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021803600000000001 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4146 3.1 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head [write 0~1129472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 1.9682774009999999, 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067860845000000003, 2024-04-20T02:52:54.263 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4146 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:54.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180199+0000", 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180439+0000", 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000239375 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180449+0000", 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0220999999999999e-05 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180456+0000", 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0090000000000001e-06 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181170+0000", 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071358900000000004 2024-04-20T02:52:54.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181331+0000", 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016178500000000001 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,5", 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181490+0000", 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158703 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225260+0000", 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043770166999999999 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226398+0000", 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001138189 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247971+0000", 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021572728999999999 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248021+0000", 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0176999999999997e-05 2024-04-20T02:52:54.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:54.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:54.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:54.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248062+0000", 2024-04-20T02:52:54.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0472000000000001e-05 2024-04-20T02:52:54.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.269 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:54.269 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:54.269 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:54.270 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:54.270 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:54.262+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:54.270 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:54.291 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:54.312 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:54.333 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:54.353 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:54.361 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:54.362+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:54.366 INFO:tasks.rados.rados.0.smithi105.stdout:1802: finishing write tid 1 to smithi10548823-5 2024-04-20T02:52:54.366 INFO:tasks.rados.rados.0.smithi105.stdout:1802: finishing write tid 2 to smithi10548823-5 2024-04-20T02:52:54.366 INFO:tasks.rados.rados.0.smithi105.stdout:1802: finishing write tid 3 to smithi10548823-5 2024-04-20T02:52:54.367 INFO:tasks.rados.rados.0.smithi105.stdout:1802: finishing write tid 4 to smithi10548823-5 2024-04-20T02:52:54.367 INFO:tasks.rados.rados.0.smithi105.stdout:1802: oid 5 updating version 0 to 212 2024-04-20T02:52:54.367 INFO:tasks.rados.rados.0.smithi105.stdout:1802: oid 5 version 212 is already newer than 211 2024-04-20T02:52:54.367 INFO:tasks.rados.rados.0.smithi105.stdout:1802: oid 5 version 212 is already newer than 210 2024-04-20T02:52:54.367 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 212 (ObjNum 368 snap 37 seq_num 368) dirty exists 2024-04-20T02:52:54.367 INFO:tasks.rados.rados.0.smithi105.stdout:1802: left oid 5 (ObjNum 368 snap 37 seq_num 368) 2024-04-20T02:52:54.367 INFO:tasks.rados.rados.0.smithi105.stdout:1803: finishing set_chunk to oid 41 2024-04-20T02:52:54.367 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 234 (ObjNum 333 snap 36 seq_num 333) dirty exists 2024-04-20T02:52:54.367 INFO:tasks.rados.rados.0.smithi105.stdout:1804: expect (ObjNum 352 snap 37 seq_num 352) 2024-04-20T02:52:54.374 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:54.405 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:54.405 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:54.407 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:54.407 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:54.407 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:54.407 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:54.407 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:54.407 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-20T02:52:54.407 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:54.407 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:54.407 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:54.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:54.417 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:54.502 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:54.503+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:54.563 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:54.564+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:54.588 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:54.589 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:54.600 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:1808: finishing rollback tid 0 to smithi10548823-44 2024-04-20T02:52:54.632 INFO:tasks.rados.rados.0.smithi105.stdout:1807: expect (ObjNum 294 snap 32 seq_num 294) 2024-04-20T02:52:54.665 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:54.665+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:54.763 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:54.764 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:54.765 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:54.766+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:54.774 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:54.854 INFO:tasks.rados.rados.0.smithi105.stdout:1805: expect (ObjNum 302 snap 32 seq_num 302) 2024-04-20T02:52:54.866 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:54.867+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:54.944 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:54.945 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:54.955 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:54.966 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:54.967+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:55.067 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:55.068+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:55.084 INFO:tasks.rados.rados.0.smithi105.stdout:1808: finishing rollback tid 1 to smithi10548823-44 2024-04-20T02:52:55.084 INFO:tasks.rados.rados.0.smithi105.stdout:1800: done (8 left) 2024-04-20T02:52:55.084 INFO:tasks.rados.rados.0.smithi105.stdout:1801: done (7 left) 2024-04-20T02:52:55.084 INFO:tasks.rados.rados.0.smithi105.stdout:1802: done (6 left) 2024-04-20T02:52:55.085 INFO:tasks.rados.rados.0.smithi105.stdout:1803: done (5 left) 2024-04-20T02:52:55.085 INFO:tasks.rados.rados.0.smithi105.stdout:1804: done (4 left) 2024-04-20T02:52:55.085 INFO:tasks.rados.rados.0.smithi105.stdout:1805: done (3 left) 2024-04-20T02:52:55.085 INFO:tasks.rados.rados.0.smithi105.stdout:1807: done (2 left) 2024-04-20T02:52:55.085 INFO:tasks.rados.rados.0.smithi105.stdout:1809: write oid 47 current snap is 37 2024-04-20T02:52:55.085 INFO:tasks.rados.rados.0.smithi105.stdout:1808: finishing rollback tid 2 to smithi10548823-44 2024-04-20T02:52:55.086 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 138 (ObjNum 208 snap 19 seq_num 208) dirty exists 2024-04-20T02:52:55.086 INFO:tasks.rados.rados.0.smithi105.stdout:1806: expect (ObjNum 226 snap 22 seq_num 226) 2024-04-20T02:52:55.126 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:55.126 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:55.136 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:55.180 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:55.180+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:55.268 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:55.268 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:55.269 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:55.269+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:55.279 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:55.370 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:55.370+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:55.402 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:52:55.403 INFO:teuthology.orchestra.run.smithi138.stdout: marking collection for removal 2024-04-20T02:52:55.403 INFO:teuthology.orchestra.run.smithi138.stdout:setting '_remove' omap key 2024-04-20T02:52:55.403 INFO:teuthology.orchestra.run.smithi138.stdout:finish_remove_pgs 2.6_head removing 2.6 2024-04-20T02:52:55.404 INFO:teuthology.orchestra.run.smithi138.stdout:Remove successful 2024-04-20T02:52:55.417 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:55.417 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:55.428 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:55.431 INFO:tasks.rados.rados.0.smithi105.stdout:1809: seq_num 369 ranges {601783=551825,1776679=425104} 2024-04-20T02:52:55.438 INFO:tasks.rados.rados.0.smithi105.stdout:1809: writing smithi10548823-47 from 601783 to 1153608 tid 1 2024-04-20T02:52:55.444 INFO:tasks.rados.rados.0.smithi105.stdout:1809: writing smithi10548823-47 from 1776679 to 2201783 tid 2 2024-04-20T02:52:55.444 INFO:tasks.rados.rados.0.smithi105.stdout:1806: done (2 left) 2024-04-20T02:52:55.444 INFO:tasks.rados.rados.0.smithi105.stdout:1808: done (1 left) 2024-04-20T02:52:55.444 INFO:tasks.rados.rados.0.smithi105.stdout:1810: read oid 14 snap 35 2024-04-20T02:52:55.444 INFO:tasks.rados.rados.0.smithi105.stdout:1810: expect (ObjNum 274 snap 26 seq_num 274) 2024-04-20T02:52:55.444 INFO:tasks.rados.rados.0.smithi105.stdout:1811: snap_remove snap 32 2024-04-20T02:52:55.470 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:55.471+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:55.560 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:55.561 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:55.571 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:55.571 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:55.571+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:55.671 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:55.671+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:55.741 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:55.741 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:55.751 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:55.772 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:55.772+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:55.873 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:55.873+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:55.880 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:55.880 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:55.890 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:55.926 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.6.4 2024-04-20T02:52:55.974 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:55.974+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:56.074 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:56.075+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:56.103 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:56.104 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:56.114 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:52:56.175 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:56.175+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:56.275 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:56.276+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:56.283 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:56.283 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:56.294 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:52:56.376 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:56.376+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:56.405 INFO:tasks.rados.rados.0.smithi105.stdout:1811: done (2 left) 2024-04-20T02:52:56.406 INFO:tasks.rados.rados.0.smithi105.stdout:1812: set_chunk oid 2 target oid 2024-04-20T02:52:56.413 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:52:56.436 INFO:tasks.rados.rados.0.smithi105.stdout:1812: set_chunk oid 2 offset: 163328 length: 1026560 target oid 694a8ccf21d1f0963fcf42cedd4644a5cfa0fc04f1bfcda0d2772f59d86ac775 offset: 0 snap -1 2024-04-20T02:52:56.437 INFO:tasks.rados.rados.0.smithi105.stdout:1813: tier_evict oid 48 2024-04-20T02:52:56.437 INFO:tasks.rados.rados.0.smithi105.stdout:1813: tier_evict oid 48 snap -1 2024-04-20T02:52:56.437 INFO:tasks.rados.rados.0.smithi105.stdout:1810: expect (ObjNum 274 snap 26 seq_num 274) 2024-04-20T02:52:56.477 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:56.478+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:56.481 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:56.481 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:56.492 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:52:56.569 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:56.578 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:56.578+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1809: finishing write tid 1 to smithi10548823-47 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1809: finishing write tid 2 to smithi10548823-47 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1809: finishing write tid 3 to smithi10548823-47 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1809: finishing write tid 4 to smithi10548823-47 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1809: oid 47 updating version 0 to 153 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1809: oid 47 updating version 153 to 154 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1809: oid 47 version 154 is already newer than 152 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 154 (ObjNum 369 snap 37 seq_num 369) dirty exists 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1809: left oid 47 (ObjNum 369 snap 37 seq_num 369) 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1809: done (3 left) 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1810: done (2 left) 2024-04-20T02:52:56.585 INFO:tasks.rados.rados.0.smithi105.stdout:1814: write (excl) oid 18 current snap is 37 2024-04-20T02:52:56.586 INFO:tasks.rados.rados.0.smithi105.stdout:1812: finishing set_chunk to oid 2 2024-04-20T02:52:56.586 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 163328 length: 1026560 2024-04-20T02:52:56.586 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 307 snap 33 seq_num 307) dirty exists 2024-04-20T02:52:56.586 INFO:tasks.rados.rados.0.smithi105.stdout:1813: got (22) Invalid argument 2024-04-20T02:52:56.586 INFO:tasks.rados.rados.0.smithi105.stdout:1814: seq_num 370 ranges {618590=627074,1835161=383429} 2024-04-20T02:52:56.588 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:56.591 INFO:tasks.rados.rados.0.smithi105.stdout:1814: writing smithi10548823-18 from 618590 to 1245664 tid 1 2024-04-20T02:52:56.598 INFO:tasks.rados.rados.0.smithi105.stdout:1814: writing smithi10548823-18 from 1835161 to 2218590 tid 2 2024-04-20T02:52:56.598 INFO:tasks.rados.rados.0.smithi105.stdout:1812: done (2 left) 2024-04-20T02:52:56.598 INFO:tasks.rados.rados.0.smithi105.stdout:1813: done (1 left) 2024-04-20T02:52:56.598 INFO:tasks.rados.rados.0.smithi105.stdout:1815: read oid 1 snap -1 2024-04-20T02:52:56.598 INFO:tasks.rados.rados.0.smithi105.stdout:1815: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:56.598 INFO:tasks.rados.rados.0.smithi105.stdout:1816: read oid 43 snap -1 2024-04-20T02:52:56.598 INFO:tasks.rados.rados.0.smithi105.stdout:1816: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:52:56.598 INFO:tasks.rados.rados.0.smithi105.stdout:1817: read oid 3 snap -1 2024-04-20T02:52:56.598 INFO:tasks.rados.rados.0.smithi105.stdout:1817: expect deleted 2024-04-20T02:52:56.598 INFO:tasks.rados.rados.0.smithi105.stdout:1818: snap_create 2024-04-20T02:52:56.601 INFO:tasks.rados.rados.0.smithi105.stdout:1816: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:52:56.611 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:56.635 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:56.655 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:56.662 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:56.662 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:52:56.662 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:52:56.662 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:52:56.662 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:56.673 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:52:56.674 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:56.679 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:56.679+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:56.694 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:56.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:56.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:56.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:56.694 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:56.695 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-20T02:52:56.695 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:56.695 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:56.695 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:56.698 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:56.780 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:56.780+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:56.805 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:56.805 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 107.355374274, 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:56.806 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.807 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.808 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.809 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.305191507000004, 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.810 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:52:56.811 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:52:56.812 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:56.813 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.814 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.305126057999999, 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:52:56.815 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:52:56.816 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:52:56.817 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.818 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 98.30510056, 2024-04-20T02:52:56.819 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.820 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.821 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.822 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.823 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2524 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1723287~549075] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 68.517591077000006, 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054106970999999997, 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2524 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.824 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286419+0000", 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.825 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286924+0000", 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050494600000000004 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286944+0000", 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9833000000000001e-05 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286957+0000", 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2901000000000001e-05 2024-04-20T02:52:56.826 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298074+0000", 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011117320999999999 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298151+0000", 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6909999999999994e-05 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298274+0000", 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012279700000000001 2024-04-20T02:52:56.827 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326267+0000", 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027993284 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331226+0000", 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049590559999999999 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340442+0000", 2024-04-20T02:52:56.828 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092155740000000007 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340488+0000", 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6208000000000002e-05 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340528+0000", 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0277000000000001e-05 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.829 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3637 4.1 e137/60 4:80c7a726:::smithi10548823-19:head v 137'124, mlcod=120'120)", 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 21.179603518, 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052640605, 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.830 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624408+0000", 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624537+0000", 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012880199999999999 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.831 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624548+0000", 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0216e-05 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624556+0000", 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6179999999999995e-06 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624603+0000", 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6601999999999999e-05 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.832 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624632+0000", 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9277000000000001e-05 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.676999+0000", 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052366639999999999 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.677049+0000", 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0822999999999998e-05 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.833 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3880 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'231, mlcod=141'228)", 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.397108537999999, 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.061291667000000001, 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:52:56.834 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406903+0000", 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407359+0000", 2024-04-20T02:52:56.835 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00045623700000000002 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407370+0000", 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1052e-05 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407377+0000", 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1080000000000004e-06 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407429+0000", 2024-04-20T02:52:56.836 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2033999999999997e-05 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407458+0000", 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8050000000000001e-05 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468161+0000", 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060703628000000003 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468196+0000", 2024-04-20T02:52:56.837 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4344999999999997e-05 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3888 4.7 4:e2c0d65b:::smithi10548823-46:head [write 413538~536345,stat] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.378555410000001, 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.059175945000000001, 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.838 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3888 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.839 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425456+0000", 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426097+0000", 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00064115099999999996 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426113+0000", 2024-04-20T02:52:56.840 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6118000000000001e-05 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426122+0000", 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.115e-06 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426179+0000", 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6815999999999999e-05 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426316+0000", 2024-04-20T02:52:56.841 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000136606 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426499+0000", 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000183191 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443652+0000", 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017153471999999999 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.842 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484036+0000", 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040384097000000001 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484231+0000", 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000194811 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.843 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484363+0000", 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013146300000000001 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484426+0000", 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3563000000000001e-05 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484633+0000", 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020662299999999999 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.844 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3889 4.7 4:e2c0d65b:::smithi10548823-46:head [write 1395221~583781] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.371661435, 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.057918795000000002, 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3889 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.845 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432350+0000", 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:56.846 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433490+0000", 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001140606 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433506+0000", 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5828000000000002e-05 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433515+0000", 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8279999999999992e-06 2024-04-20T02:52:56.847 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443692+0000", 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010177222 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443839+0000", 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014662 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.444072+0000", 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023362200000000001 2024-04-20T02:52:56.848 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484114+0000", 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040041641000000003 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484244+0000", 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012987200000000001 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484279+0000", 2024-04-20T02:52:56.849 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5301e-05 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490055+0000", 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057763459999999999 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490115+0000", 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9320999999999997e-05 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490270+0000", 2024-04-20T02:52:56.850 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015498199999999999 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3890 4.7 4:e2c0d65b:::smithi10548823-46:head [write 2556103~658617] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.363647333999999, 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066878325000000002, 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.851 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3890 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.852 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440363+0000", 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441831+0000", 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001467632 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441842+0000", 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0540000000000001e-05 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.853 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441850+0000", 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7260000000000008e-06 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463536+0000", 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02168544 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463681+0000", 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000145086 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:56.854 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463941+0000", 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00025968600000000002 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484317+0000", 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020376452 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490318+0000", 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060012490000000002 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.855 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490398+0000", 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9598000000000006e-05 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506988+0000", 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016589706999999999 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507066+0000", 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8299000000000004e-05 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.856 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507243+0000", 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000177503 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3891 4.7 4:e2c0d65b:::smithi10548823-46:head [setxattr _header (57) in=7b,truncate 3613538] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.362151825, 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088590031, 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.857 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3891 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.858 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441859+0000", 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441864+0000", 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8069999999999999e-06 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441869+0000", 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0989999999999998e-06 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.859 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441875+0000", 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5820000000000001e-06 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484675+0000", 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042799867999999998 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484817+0000", 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014236899999999999 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:52:56.860 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485048+0000", 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023057199999999999 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490359+0000", 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053116819999999999 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506963+0000", 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016603562999999998 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.861 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507256+0000", 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00029288699999999998 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530316+0000", 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023060343000000001 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530356+0000", 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9301999999999997e-05 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.862 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530451+0000", 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5037999999999994e-05 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3892 4.7 4:e2c0d65b:::smithi10548823-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.362132137, 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088912482000000001, 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.863 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3892 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.864 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441883+0000", 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9579999999999999e-06 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441889+0000", 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7499999999999997e-06 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.865 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441895+0000", 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485431+0000", 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043536321000000003 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485537+0000", 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010612599999999999 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.866 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530489+0000", 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044951996000000001 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530598+0000", 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108937 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530793+0000", 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019441 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.867 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3898 4.8 4:1c68e542:::smithi10548823-8:head [write 447152~403196] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.337789054, 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.056917473000000003, 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3898 2024-04-20T02:52:56.868 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466222+0000", 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:56.869 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466722+0000", 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00049941299999999998 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466734+0000", 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2860999999999999e-05 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466743+0000", 2024-04-20T02:52:56.870 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7159999999999992e-06 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466809+0000", 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6165000000000001e-05 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466928+0000", 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011869199999999999 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.467158+0000", 2024-04-20T02:52:56.871 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000230251 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498338+0000", 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.03117996 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.522958+0000", 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024619318000000001 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.872 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523002+0000", 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4675999999999997e-05 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523082+0000", 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9803999999999997e-05 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523141+0000", 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8767000000000002e-05 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.873 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3899 4.8 4:1c68e542:::smithi10548823-8:head [write 1512185~587450] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 12.329861287, 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060518556000000001, 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3899 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.874 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.875 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474149+0000", 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474686+0000", 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053696199999999999 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474703+0000", 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.615e-05 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.876 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474713+0000", 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0829000000000001e-05 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478533+0000", 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003819687 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478640+0000", 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106974 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:56.877 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478766+0000", 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125859 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523803+0000", 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045036616000000002 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526432+0000", 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026291489999999999 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.878 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534589+0000", 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081572209999999992 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534623+0000", 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534670+0000", 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6968999999999997e-05 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.879 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4016 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 665643~697786] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.5426925219999994, 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060676251, 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4016 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.880 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261318+0000", 2024-04-20T02:52:56.881 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262081+0000", 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076285299999999997 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262095+0000", 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3986e-05 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262118+0000", 2024-04-20T02:52:56.882 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2407000000000001e-05 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262164+0000", 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6248999999999997e-05 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262238+0000", 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4202000000000005e-05 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262363+0000", 2024-04-20T02:52:56.883 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012437799999999999 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314944+0000", 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052581687000000002 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317407+0000", 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024628300000000001 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321887+0000", 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0044803020000000002 2024-04-20T02:52:56.884 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321941+0000", 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3769e-05 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321996+0000", 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4988000000000003e-05 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.885 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4017 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 2039745~753122] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 8.5350627239999994, 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053146383999999998, 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4017 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.886 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268948+0000", 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.887 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269678+0000", 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072975799999999995 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269693+0000", 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.453e-05 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269704+0000", 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1623e-05 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.888 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277190+0000", 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074859849999999997 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277276+0000", 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5809999999999993e-05 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277405+0000", 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012914600000000001 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.889 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314952+0000", 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037546931999999998 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317441+0000", 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024889840000000001 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322031+0000", 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004589465 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.890 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322059+0000", 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322096+0000", 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591999999999998e-05 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4074 4.11 4:8b91c759:::smithi10548823-5:head [write 1591248~408851] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.6620953350000001, 2024-04-20T02:52:56.891 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053876146, 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4074 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.892 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141916+0000", 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142740+0000", 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000823616 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.893 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142750+0000", 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6150000000000003e-06 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142756+0000", 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9419999999999998e-06 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:52:56.894 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2061999999999997e-05 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142815+0000", 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5050000000000004e-06 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174069+0000", 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031253933999999997 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174193+0000", 2024-04-20T02:52:56.895 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012386 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174438+0000", 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00024561400000000001 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187472+0000", 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013033375999999999 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.188278+0000", 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00080595500000000002 2024-04-20T02:52:56.896 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195551+0000", 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072733340000000002 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195631+0000", 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9882999999999994e-05 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195793+0000", 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000162223 2024-04-20T02:52:56.897 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4075 4.11 4:8b91c759:::smithi10548823-5:head [setxattr _header (56) in=7b,truncate 2000099] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.6612444029999995, 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084064655000000002, 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.898 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4075 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.899 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142767+0000", 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142778+0000", 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0709e-05 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142783+0000", 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4539999999999998e-06 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.900 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142788+0000", 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2850000000000002e-06 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142845+0000", 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6919000000000002e-05 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142856+0000", 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0936e-05 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.901 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186635+0000", 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043778897999999997 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186756+0000", 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121087 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186967+0000", 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021064599999999999 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.902 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.189477+0000", 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025104929999999999 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.190395+0000", 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00091717399999999998 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226595+0000", 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036200195999999997 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.903 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226661+0000", 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6465999999999997e-05 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226833+0000", 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000171471 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4076 4.11 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:52:56.904 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.6612033969999995, 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084363853000000003, 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4076 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.905 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142808+0000", 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142811+0000", 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9229999999999998e-06 2024-04-20T02:52:56.906 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142818+0000", 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4420000000000003e-06 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142823+0000", 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5199999999999997e-06 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142875+0000", 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1759999999999999e-05 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.907 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142880+0000", 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6639999999999997e-06 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187624+0000", 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044743949999999998 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187726+0000", 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0001022 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.908 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226882+0000", 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039155862 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227003+0000", 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120715 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227173+0000", 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017032899999999999 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.909 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4183 3.2 e154/49 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head v 154'234, mlcod=154'232)", 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 3.1510744640000001, 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.063121930000000007, 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.910 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652936+0000", 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657053+0000", 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041172520000000001 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.911 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657067+0000", 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3865e-05 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657076+0000", 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5390000000000007e-06 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657121+0000", 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5114000000000002e-05 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.912 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657138+0000", 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7600000000000001e-05 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716021+0000", 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.058882663000000002 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716060+0000", 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8826999999999999e-05 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.913 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:56.914 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:52:56.914 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:56.915 INFO:tasks.rados.rados.0.smithi105.stdout:1815: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:52:56.915 INFO:tasks.rados.rados.0.smithi105.stdout:1814: finishing write tid 1 to smithi10548823-18 2024-04-20T02:52:56.915 INFO:tasks.rados.rados.0.smithi105.stdout:1814: finishing write tid 2 to smithi10548823-18 2024-04-20T02:52:56.915 INFO:tasks.rados.rados.0.smithi105.stdout:1814: finishing write tid 3 to smithi10548823-18 2024-04-20T02:52:56.915 INFO:tasks.rados.rados.0.smithi105.stdout:1814: finishing write tid 4 to smithi10548823-18 2024-04-20T02:52:56.915 INFO:tasks.rados.rados.0.smithi105.stdout:1814: oid 18 updating version 0 to 140 2024-04-20T02:52:56.915 INFO:tasks.rados.rados.0.smithi105.stdout:1814: oid 18 updating version 140 to 141 2024-04-20T02:52:56.915 INFO:tasks.rados.rados.0.smithi105.stdout:1814: oid 18 version 141 is already newer than 139 2024-04-20T02:52:56.915 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 18 v 141 (ObjNum 370 snap 37 seq_num 370) dirty exists 2024-04-20T02:52:56.915 INFO:tasks.rados.rados.0.smithi105.stdout:1814: left oid 18 (ObjNum 370 snap 37 seq_num 370) 2024-04-20T02:52:56.916 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:52:56.917 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:56.915+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:57.016 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:57.016+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:57.047 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:57.047 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:57.047 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:57.057 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:52:57.074 INFO:teuthology.orchestra.run.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e149 pg_nums {1={14=1},2={18=8},3={21=4},4={22=16,60=26,81=25,85=24,89=23,96=26,110=25,114=24,118=23,122=22,126=21}} deleted_pools ) 2024-04-20T02:52:57.074 INFO:teuthology.orchestra.run.smithi138.stdout:Importing pgid 2.6 2024-04-20T02:52:57.074 INFO:teuthology.orchestra.run.smithi138.stdout:write_pg epoch 140 info 2.6( empty local-lis/les=18/19 n=0 ec=18/18 lis/c=18/18 les/c/f=19/19/0 sis=18) 2024-04-20T02:52:57.078 INFO:teuthology.orchestra.run.smithi138.stdout:Import successful 2024-04-20T02:52:57.116 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:57.117+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:57.217 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:57.217+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:57.226 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:57.226 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:57.226 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:57.226 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:57.226 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:57.236 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:52:57.321 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:57.321+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:57.403 INFO:tasks.rados.rados.0.smithi105.stdout:1814: done (4 left) 2024-04-20T02:52:57.404 INFO:tasks.rados.rados.0.smithi105.stdout:1815: done (3 left) 2024-04-20T02:52:57.404 INFO:tasks.rados.rados.0.smithi105.stdout:1816: done (2 left) 2024-04-20T02:52:57.404 INFO:tasks.rados.rados.0.smithi105.stdout:1817: done (1 left) 2024-04-20T02:52:57.404 INFO:tasks.rados.rados.0.smithi105.stdout:1818: done (0 left) 2024-04-20T02:52:57.404 INFO:tasks.rados.rados.0.smithi105.stdout:1819: write oid 6 current snap is 38 2024-04-20T02:52:57.404 INFO:tasks.rados.rados.0.smithi105.stdout:1819: seq_num 371 ranges {635397=702323,1893643=341754} 2024-04-20T02:52:57.406 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:57.406 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:57.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:57.406 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-04-20T02:52:57.407 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:57.410 INFO:tasks.rados.rados.0.smithi105.stdout:1819: writing smithi10548823-6 from 635397 to 1337720 tid 1 2024-04-20T02:52:57.415 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:52:57.416 INFO:tasks.rados.rados.0.smithi105.stdout:1819: writing smithi10548823-6 from 1893643 to 2235397 tid 2 2024-04-20T02:52:57.418 INFO:tasks.rados.rados.0.smithi105.stdout:1820: read oid 49 snap -1 2024-04-20T02:52:57.418 INFO:tasks.rados.rados.0.smithi105.stdout:1820: expect (ObjNum 352 snap 37 seq_num 352) 2024-04-20T02:52:57.418 INFO:tasks.rados.rados.0.smithi105.stdout:1821: read oid 20 snap 35 2024-04-20T02:52:57.418 INFO:tasks.rados.rados.0.smithi105.stdout:1821: expect (ObjNum 326 snap 35 seq_num 326) 2024-04-20T02:52:57.418 INFO:tasks.rados.rados.0.smithi105.stdout:1822: read oid 36 snap -1 2024-04-20T02:52:57.418 INFO:tasks.rados.rados.0.smithi105.stdout:1822: expect (ObjNum 360 snap 37 seq_num 360) 2024-04-20T02:52:57.418 INFO:tasks.rados.rados.0.smithi105.stdout:1823: read oid 33 snap -1 2024-04-20T02:52:57.418 INFO:tasks.rados.rados.0.smithi105.stdout:1823: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:52:57.418 INFO:tasks.rados.rados.0.smithi105.stdout:1824: read oid 24 snap -1 2024-04-20T02:52:57.419 INFO:tasks.rados.rados.0.smithi105.stdout:1824: expect (ObjNum 355 snap 37 seq_num 355) 2024-04-20T02:52:57.419 INFO:tasks.rados.rados.0.smithi105.stdout:1825: read oid 37 snap -1 2024-04-20T02:52:57.419 INFO:tasks.rados.rados.0.smithi105.stdout:1825: expect (ObjNum 327 snap 35 seq_num 327) 2024-04-20T02:52:57.419 INFO:tasks.rados.rados.0.smithi105.stdout:1826: tier_promote oid 35 2024-04-20T02:52:57.419 INFO:tasks.rados.rados.0.smithi105.stdout:1827: write oid 3 current snap is 38 2024-04-20T02:52:57.419 INFO:tasks.rados.rados.0.smithi105.stdout:1827: seq_num 372 ranges {652204=777572,1952125=300079} 2024-04-20T02:52:57.428 INFO:tasks.rados.rados.0.smithi105.stdout:1827: writing smithi10548823-3 from 652204 to 1429776 tid 1 2024-04-20T02:52:57.428 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:52:57.430 INFO:tasks.rados.rados.0.smithi105.stdout:1827: writing smithi10548823-3 from 1952125 to 2252204 tid 2 2024-04-20T02:52:57.431 INFO:tasks.rados.rados.0.smithi105.stdout:1828: read oid 2 snap -1 2024-04-20T02:52:57.431 INFO:tasks.rados.rados.0.smithi105.stdout:1828: expect (ObjNum 307 snap 33 seq_num 307) 2024-04-20T02:52:57.431 INFO:tasks.rados.rados.0.smithi105.stdout:1829: snap_remove snap 37 2024-04-20T02:52:57.435 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:57.428+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:57.521 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:57.522+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:57.585 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:57.585 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:57.586 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:57.586 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:57.596 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f /var/log/ceph/exp.2.6.4 2024-04-20T02:52:57.597 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:52:57.641 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:52:57.641+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:52:57.665 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.4 2024-04-20T02:52:57.665 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-4 show-label 2024-04-20T02:52:57.723 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:57.723+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:57.725 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:57.725 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:57.726 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:57.726 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:57.726 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:57.736 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:52:57.748 INFO:teuthology.orchestra.run.smithi138.stdout:inferring bluefs devices from bluestore path 2024-04-20T02:52:57.749 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:52:57.749 INFO:teuthology.orchestra.run.smithi138.stdout: "/var/lib/ceph/osd/ceph-4/block": { 2024-04-20T02:52:57.749 INFO:teuthology.orchestra.run.smithi138.stdout: "osd_uuid": "1d41da44-d894-40ac-acaa-aa0df15296c6", 2024-04-20T02:52:57.749 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 96636764160, 2024-04-20T02:52:57.749 INFO:teuthology.orchestra.run.smithi138.stdout: "btime": "2024-04-20T02:49:47.177483+0000", 2024-04-20T02:52:57.749 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "main", 2024-04-20T02:52:57.749 INFO:teuthology.orchestra.run.smithi138.stdout: "bfm_blocks": "23592960", 2024-04-20T02:52:57.749 INFO:teuthology.orchestra.run.smithi138.stdout: "bfm_blocks_per_key": "128", 2024-04-20T02:52:57.749 INFO:teuthology.orchestra.run.smithi138.stdout: "bfm_bytes_per_block": "4096", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "bfm_size": "96636764160", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "bluefs": "1", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_fsid": "a524a365-eaec-4ebd-aa4d-0d76d55a252f", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "ceph_version_when_created": "ceph version 18.2.2-1252-g15eecde2 (15eecde2bab9803889d929c7e423d07970d8cbed) reef (stable)", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "created_at": "2024-04-20T02:49:49.511996Z", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "kv_backend": "rocksdb", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "magic": "ceph osd volume v026", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "mkfs_done": "yes", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "ready": "ready", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "require_osd_release": "18", 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: "whoami": "4" 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:52:57.750 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:52:57.751 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.4 2024-04-20T02:52:57.751 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-4 show-sharding 2024-04-20T02:52:57.824 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:57.824+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:57.906 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 131.36110643699999, 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:52:57.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 126.973183976, 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:52:57.911 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:52:57.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:52:57.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:52:57.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:52:57.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:52:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:52:57.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.917 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:57.918 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:57.918 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 126.97292376999999, 2024-04-20T02:52:57.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:52:57.918 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.919 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.919 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.919 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.919 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.919 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:52:57.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.919 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:57.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:52:57.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:52:57.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:52:57.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:52:57.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:52:57.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:52:57.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:52:57.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:52:57.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:57.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:52:57.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:52:57.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.927 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:52:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 126.97290230199999, 2024-04-20T02:52:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:52:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.928 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.929 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.929 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:52:57.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.929 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:57.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:57.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:52:57.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:57.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:52:57.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:52:57.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.934 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.934 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.934 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:52:57.934 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 117.77923036999999, 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:52:57.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:52:57.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2292 4.3 4:c07c6daa:::smithi10548823-7:head [write 544593~799729] snapc 1e=[1e,1d,1b,18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 83.746187178, 2024-04-20T02:52:57.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061161747000000002, 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2292 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159577+0000", 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160019+0000", 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000442674 2024-04-20T02:52:57.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160045+0000", 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5446e-05 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160058+0000", 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3444e-05 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160487+0000", 2024-04-20T02:52:57.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000428899 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160616+0000", 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012861099999999999 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160838+0000", 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000222415 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207115+0000", 2024-04-20T02:52:57.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046276543000000003 2024-04-20T02:52:57.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207164+0000", 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9497999999999999e-05 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220657+0000", 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013492386 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220698+0000", 2024-04-20T02:52:57.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1451e-05 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220741+0000", 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3078000000000003e-05 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2699 4.0 e95/60 4:007b2a87:::smithi10548823-39:head v 95'100, mlcod=94'97)", 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.55660589, 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061625529999999998, 2024-04-20T02:52:57.947 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349160+0000", 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349351+0000", 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019061600000000001 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349364+0000", 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2835e-05 2024-04-20T02:52:57.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349376+0000", 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349466+0000", 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9615000000000002e-05 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349492+0000", 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6494000000000001e-05 2024-04-20T02:52:57.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.410234+0000", 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060741334000000001 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.410786+0000", 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000552539 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3423 4.3 4:c07c6daa:::smithi10548823-7:head [write 558065~619869] snapc 3a=[3a,39,38,36,35,33] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 29.516318155, 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061409060000000001, 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3423 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389446+0000", 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389586+0000", 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014017899999999999 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389605+0000", 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389617+0000", 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1956999999999999e-05 2024-04-20T02:52:57.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.412832+0000", 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023215612 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.412909+0000", 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6555000000000001e-05 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:57.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.413035+0000", 2024-04-20T02:52:57.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000126599 2024-04-20T02:52:57.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450700+0000", 2024-04-20T02:52:57.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037664717 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450712+0000", 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1422e-05 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450741+0000", 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9791999999999999e-05 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450801+0000", 2024-04-20T02:52:57.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9199e-05 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450857+0000", 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6882999999999997e-05 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3771 4.b 4:d4308fef:::smithi10548823-6:head [write 2548663~763022] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.723873645000001, 2024-04-20T02:52:57.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072733780999999997, 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3771 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181891+0000", 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182301+0000", 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040985099999999998 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182317+0000", 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6305999999999999e-05 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182329+0000", 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1337e-05 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198720+0000", 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016391606 2024-04-20T02:52:57.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198807+0000", 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7319999999999997e-05 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198925+0000", 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117345 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218062+0000", 2024-04-20T02:52:57.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019137134 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234990+0000", 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016928367 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254488+0000", 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019497460000000001 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254571+0000", 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3029999999999996e-05 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254627+0000", 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5865000000000001e-05 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3772 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3395047] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.723427073, 2024-04-20T02:52:57.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072419708999999999, 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3772 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182343+0000", 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0099999999999997e-06 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182349+0000", 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9569999999999999e-06 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182354+0000", 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.826e-06 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217228+0000", 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034874555000000002 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217312+0000", 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3478000000000003e-05 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217431+0000", 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119512 2024-04-20T02:52:57.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234969+0000", 2024-04-20T02:52:57.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017537833999999999 2024-04-20T02:52:57.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235012+0000", 2024-04-20T02:52:57.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3411000000000002e-05 2024-04-20T02:52:57.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254638+0000", 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019625815000000001 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254682+0000", 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3945e-05 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254759+0000", 2024-04-20T02:52:57.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6882e-05 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3773 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.723408465999999, 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072538680999999994, 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.971 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:57.973 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.973 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.973 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.973 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3773 2024-04-20T02:52:57.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.973 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:57.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182360+0000", 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3309999999999998e-06 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182366+0000", 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.981e-06 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182371+0000", 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5390000000000001e-06 2024-04-20T02:52:57.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217932+0000", 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035561700000000002 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218020+0000", 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7352e-05 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254778+0000", 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036757971 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254819+0000", 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1525e-05 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254897+0000", 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7621999999999996e-05 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.977 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3774 4.3 4:c07c6daa:::smithi10548823-7:head [write 611854~717004] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.190321+0000", 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.715445313, 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064562913999999999, 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.978 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3774 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190321+0000", 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190321+0000", 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190319+0000", 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190704+0000", 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038458399999999999 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190721+0000", 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6988000000000001e-05 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190734+0000", 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.276e-05 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217683+0000", 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026949832 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217775+0000", 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1669000000000006e-05 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217991+0000", 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021635400000000001 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254534+0000", 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036542842999999998 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254573+0000", 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8921000000000002e-05 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254780+0000", 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020691700000000001 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254833+0000", 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2785999999999999e-05 2024-04-20T02:52:57.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254884+0000", 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1159999999999998e-05 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3775 4.3 4:c07c6daa:::smithi10548823-7:head [write 2126143~561767] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.198678+0000", 2024-04-20T02:52:57.984 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.707088053, 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062905904999999998, 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3775 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198678+0000", 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198678+0000", 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198677+0000", 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198991+0000", 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031458700000000002 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.199005+0000", 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3134e-05 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.199013+0000", 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3489999999999992e-06 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234979+0000", 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035965991000000003 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235102+0000", 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012322400000000001 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235329+0000", 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022682799999999999 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254591+0000", 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019262029999999999 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254607+0000", 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6150999999999998e-05 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.261494+0000", 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068870779999999996 2024-04-20T02:52:57.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.261539+0000", 2024-04-20T02:52:57.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4400000000000002e-05 2024-04-20T02:52:57.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.261584+0000", 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5775000000000001e-05 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3880 4.3 4:c32272c0:::smithi10548823-41:head [write 796731~461096] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.406265+0000", 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.499501548, 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066110050000000004, 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.991 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3880 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406265+0000", 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406265+0000", 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406264+0000", 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406481+0000", 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021666200000000001 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:57.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406493+0000", 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406499+0000", 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3609999999999997e-06 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406540+0000", 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0728e-05 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406632+0000", 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1821000000000001e-05 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406742+0000", 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110178 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.459209+0000", 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.052467645 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.459342+0000", 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013213 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.472265+0000", 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012923128000000001 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.472321+0000", 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.63e-05 2024-04-20T02:52:57.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.472375+0000", 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4042999999999997e-05 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4057 4.b 4:d4308fef:::smithi10548823-6:head [write 749678~790384,stat] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.7625537260000002, 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074563737000000005, 2024-04-20T02:52:57.997 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4057 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143210+0000", 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143760+0000", 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055015999999999995 2024-04-20T02:52:57.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143772+0000", 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1345000000000001e-05 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143781+0000", 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0929999999999992e-06 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143895+0000", 2024-04-20T02:52:58.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113998 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143906+0000", 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1195999999999999e-05 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.153590+0000", 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096836000000000005 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:52:58.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155611+0000", 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002021012 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217691+0000", 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062080244 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217777+0000", 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5583999999999999e-05 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.002 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4058 4.b 4:d4308fef:::smithi10548823-6:head [write 2332155~504118] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.7619727000000003, 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074090323999999999, 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4058 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.003 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143793+0000", 2024-04-20T02:52:58.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144104+0000", 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000310971 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144112+0000", 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1990000000000006e-06 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144118+0000", 2024-04-20T02:52:58.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.443e-06 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144162+0000", 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3066999999999999e-05 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144172+0000", 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0083e-05 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155911+0000", 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011739657000000001 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155993+0000", 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1754000000000004e-05 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217810+0000", 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061816958999999998 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217884+0000", 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4098999999999996e-05 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4059 4.b 4:d4308fef:::smithi10548823-6:head [write 3271857~549469] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.7616365389999995, 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073860173000000001, 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.008 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4059 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144129+0000", 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144439+0000", 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030924000000000001 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144447+0000", 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8930000000000005e-06 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144455+0000", 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1410000000000005e-06 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144505+0000", 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0698000000000001e-05 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144520+0000", 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4605e-05 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156034+0000", 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011513809 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156130+0000", 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6039999999999995e-05 2024-04-20T02:52:58.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217915+0000", 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061785501999999999 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217990+0000", 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4770000000000004e-05 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4060 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3949678] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.7613011600000004, 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085188106999999999, 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4060 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144469+0000", 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.946e-06 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144476+0000", 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7909999999999999e-06 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144482+0000", 2024-04-20T02:52:58.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.601e-06 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144803+0000", 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032103699999999998 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144813+0000", 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0183999999999999e-05 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156168+0000", 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011354523 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156254+0000", 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5706999999999998e-05 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218023+0000", 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061769566999999997 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218137+0000", 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113912 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218377+0000", 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023941000000000001 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225107+0000", 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067304510000000001 2024-04-20T02:52:58.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225121+0000", 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4341e-05 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229481+0000", 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043599880000000004 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229522+0000", 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0224e-05 2024-04-20T02:52:58.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229653+0000", 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013190500000000001 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4061 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.7611615430000001, 2024-04-20T02:52:58.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085431572999999997, 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4061 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144604+0000", 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144608+0000", 2024-04-20T02:52:58.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1080000000000003e-06 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144618+0000", 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5149999999999995e-06 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144625+0000", 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8260000000000004e-06 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144846+0000", 2024-04-20T02:52:58.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022160699999999999 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144854+0000", 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0399999999999993e-06 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156327+0000", 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011472178 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:52:58.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156457+0000", 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013012599999999999 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218704+0000", 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062247080000000003 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218795+0000", 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1391000000000006e-05 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229703+0000", 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010907834999999999 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229818+0000", 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115423 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.230037+0000", 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021803600000000001 2024-04-20T02:52:58.027 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4146 3.1 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head [write 0~1129472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 5.7255655660000002, 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067860845000000003, 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.028 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4146 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180199+0000", 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180439+0000", 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000239375 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180449+0000", 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0220999999999999e-05 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180456+0000", 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0090000000000001e-06 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181170+0000", 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071358900000000004 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181331+0000", 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016178500000000001 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,5", 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181490+0000", 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158703 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225260+0000", 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043770166999999999 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226398+0000", 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001138189 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247971+0000", 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021572728999999999 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248021+0000", 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0176999999999997e-05 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248062+0000", 2024-04-20T02:52:58.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0472000000000001e-05 2024-04-20T02:52:58.034 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.034 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.034 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.034 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.034 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.034 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:58.036 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:52:58.036 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:58.036+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:58.137 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:58.137+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:58.206 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:58.206 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:58.206 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:58.206 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:58.216 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:52:58.238 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:58.238+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:58.338 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:58.338+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:58.384 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:58.384 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:58.384 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:52:58.385 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:52:58.385 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:58.394 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:52:58.407 INFO:tasks.rados.rados.0.smithi105.stdout:1829: done (10 left) 2024-04-20T02:52:58.407 INFO:tasks.rados.rados.0.smithi105.stdout:1830: write oid 23 current snap is 38 2024-04-20T02:52:58.407 INFO:tasks.rados.rados.0.smithi105.stdout:1830: seq_num 373 ranges {669011=569174,2010607=258404} 2024-04-20T02:52:58.410 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:52:58.418 INFO:tasks.rados.rados.0.smithi105.stdout:1830: writing smithi10548823-23 from 669011 to 1238185 tid 1 2024-04-20T02:52:58.420 INFO:tasks.rados.rados.0.smithi105.stdout:1830: writing smithi10548823-23 from 2010607 to 2269011 tid 2 2024-04-20T02:52:58.420 INFO:tasks.rados.rados.0.smithi105.stdout:1831: read oid 34 snap -1 2024-04-20T02:52:58.420 INFO:tasks.rados.rados.0.smithi105.stdout:1831: expect (ObjNum 319 snap 34 seq_num 319) 2024-04-20T02:52:58.420 INFO:tasks.rados.rados.0.smithi105.stdout:1832: read oid 19 snap -1 2024-04-20T02:52:58.420 INFO:tasks.rados.rados.0.smithi105.stdout:1832: expect (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:52:58.420 INFO:tasks.rados.rados.0.smithi105.stdout:1833: read oid 43 snap -1 2024-04-20T02:52:58.420 INFO:tasks.rados.rados.0.smithi105.stdout:1833: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:52:58.420 INFO:tasks.rados.rados.0.smithi105.stdout:1834: tier_flush oid 50 2024-04-20T02:52:58.421 INFO:tasks.rados.rados.0.smithi105.stdout:1834: tier_flush oid 50 snap -1 2024-04-20T02:52:58.421 INFO:tasks.rados.rados.0.smithi105.stdout:1835: tier_promote oid 8 2024-04-20T02:52:58.421 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:52:58.421 INFO:tasks.rados.rados.0.smithi105.stdout:1820: expect (ObjNum 352 snap 37 seq_num 352) 2024-04-20T02:52:58.440 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:58.440+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:58.540 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:58.541+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:58.552 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:58.552 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:52:58.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:52:58.552 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:52:58.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.552 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:58.552 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:58.552 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 131.965836266, 2024-04-20T02:52:58.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:52:58.552 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 131.945106955, 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.558 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.563 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 131.93585520400001, 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.564 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:52:58.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:52:58.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:52:58.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:52:58.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 124.138855988, 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 124.1328512, 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4029 4.13 4:cf11ccc0:::smithi10548823-4:head [write 699257~564637,stat] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9953086839999994, 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.86467550400000004, 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:58.578 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4029 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555666+0000", 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555809+0000", 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014306900000000001 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555832+0000", 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3680999999999999e-05 2024-04-20T02:52:58.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555846+0000", 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3654e-05 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555913+0000", 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6736999999999995e-05 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556058+0000", 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014538100000000001 2024-04-20T02:52:58.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556392+0000", 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033393099999999998 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578661+0000", 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022268553999999999 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578702+0000", 2024-04-20T02:52:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1533999999999999e-05 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578747+0000", 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4858999999999999e-05 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578811+0000", 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3520000000000002e-05 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605743+0000", 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026932418999999999 2024-04-20T02:52:58.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420312+0000", 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81456849600000003 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420343+0000", 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1911e-05 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.584 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4030 4.13 4:cf11ccc0:::smithi10548823-4:head [write 1873062~640438] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9883955629999992, 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85780235299999996, 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4030 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:52:58.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562579+0000", 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562714+0000", 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013475 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562728+0000", 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3516000000000001e-05 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562736+0000", 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5690000000000005e-06 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578835+0000", 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016099156 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578957+0000", 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121094 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.579123+0000", 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016647100000000001 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605727+0000", 2024-04-20T02:52:58.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026603643999999999 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605749+0000", 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2070999999999999e-05 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605799+0000", 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0417000000000002e-05 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605841+0000", 2024-04-20T02:52:58.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2344e-05 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605907+0000", 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5135999999999994e-05 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420373+0000", 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81446675899999998 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420383+0000", 2024-04-20T02:52:58.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0108999999999999e-05 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4031 4.13 4:cf11ccc0:::smithi10548823-4:head [write 3305941~593316] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9808339880000005, 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85027709799999995, 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.591 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4031 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570142+0000", 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570277+0000", 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135606 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570288+0000", 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0719e-05 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570294+0000", 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2519999999999996e-06 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592786+0000", 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022491220999999999 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592911+0000", 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012546599999999999 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.593118+0000", 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000207011 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605878+0000", 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012760298999999999 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605934+0000", 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5813999999999998e-05 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605963+0000", 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9275e-05 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606612+0000", 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00064813999999999996 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613782+0000", 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071705670000000001 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420411+0000", 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.80662907399999995 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420420+0000", 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5790000000000004e-06 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4032 4.13 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (56) in=7b,truncate 3899257] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:52:58.597 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9806775600000002, 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85015831100000006, 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4032 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570302+0000", 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1700000000000001e-06 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570305+0000", 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.976e-06 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570308+0000", 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8899999999999999e-06 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605985+0000", 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035677121999999999 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606066+0000", 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0797000000000006e-05 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606211+0000", 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014569999999999999 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607013+0000", 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00080160599999999995 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607046+0000", 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2938999999999998e-05 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607086+0000", 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0203000000000001e-05 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607157+0000", 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0621e-05 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613833+0000", 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066762269999999999 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420448+0000", 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.806615478 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420457+0000", 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9779999999999994e-06 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.603 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4033 4.13 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.980665686, 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85021192700000003, 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4033 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.604 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570312+0000", 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419999999999999e-06 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570316+0000", 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3730000000000001e-06 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570318+0000", 2024-04-20T02:52:58.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6680000000000001e-06 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606541+0000", 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036222797000000001 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606598+0000", 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6629999999999998e-05 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420515+0000", 2024-04-20T02:52:58.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81391740099999998 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420523+0000", 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6799999999999993e-06 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4034 4.13 4:cfd1e4d4:::smithi10548823-9:head [rollback 3f] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9806560480000002, 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85017147900000001, 2024-04-20T02:52:58.608 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4034 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570320+0000", 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570322+0000", 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.37e-06 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570325+0000", 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7800000000000001e-06 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570328+0000", 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5379999999999998e-06 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606632+0000", 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036304075999999998 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606716+0000", 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3689000000000005e-05 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420484+0000", 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81376812700000001 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420492+0000", 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1179999999999992e-06 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4062 4.9 4:9c87d36d:::smithi10548823-24:head [write 766485~465633] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.4008022669999995, 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075029242999999995, 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4062 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150173+0000", 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150435+0000", 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000262064 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150445+0000", 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8160000000000005e-06 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150450+0000", 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1880000000000001e-06 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150484+0000", 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150490+0000", 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6810000000000003e-06 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161686+0000", 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011196187 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161840+0000", 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015389000000000001 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.162103+0000", 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026333299999999998 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224927+0000", 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062823458999999998 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224993+0000", 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6667000000000004e-05 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225059+0000", 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6117000000000004e-05 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225119+0000", 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9342000000000002e-05 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225204+0000", 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4844000000000003e-05 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4063 4.9 4:9c87d36d:::smithi10548823-24:head [write 1990637~647776] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.4005195930000003, 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075097435000000004, 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.619 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4063 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150746+0000", 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028951399999999999 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150753+0000", 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-04-20T02:52:58.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150757+0000", 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2300000000000002e-06 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150801+0000", 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3421000000000003e-05 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150811+0000", 2024-04-20T02:52:58.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0581000000000001e-05 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177065+0000", 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02625367 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177164+0000", 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8900999999999994e-05 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177319+0000", 2024-04-20T02:52:58.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000155437 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225048+0000", 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047728857 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225217+0000", 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168395 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225375+0000", 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157837 2024-04-20T02:52:58.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225437+0000", 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2824999999999996e-05 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225555+0000", 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117169 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.625 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4064 4.9 4:9c87d36d:::smithi10548823-24:head [write 3299280~667205] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.40021284, 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112222372, 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4064 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150763+0000", 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151076+0000", 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000312355 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151083+0000", 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1010000000000003e-06 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151091+0000", 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0409999999999998e-06 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151127+0000", 2024-04-20T02:52:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151133+0000", 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1099999999999999e-06 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200197+0000", 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049064000000000003 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200315+0000", 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117855 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200545+0000", 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022990700000000001 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225595+0000", 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025050224999999999 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262670+0000", 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075460999999997 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262767+0000", 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6479999999999998e-05 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262835+0000", 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999999e-05 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262986+0000", 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015129799999999999 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4065 4.9 4:9c87d36d:::smithi10548823-24:head [setxattr _header (57) in=7b,truncate 3966485] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.3998729599999997, 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112193353, 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.632 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4065 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:52:58.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151103+0000", 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151107+0000", 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151111+0000", 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2309999999999998e-06 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151116+0000", 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4850000000000003e-06 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151168+0000", 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2673000000000002e-05 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151179+0000", 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0112000000000001e-05 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225611+0000", 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074432549000000001 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225689+0000", 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7612000000000002e-05 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:52:58.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225858+0000", 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016945400000000001 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262650+0000", 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036792126000000001 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262678+0000", 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8132000000000001e-05 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263077+0000", 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039868299999999999 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263141+0000", 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3448999999999995e-05 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263297+0000", 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015649699999999999 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.638 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4066 4.9 4:9c87d36d:::smithi10548823-24:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.3998561040000004, 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112663448, 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4066 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151120+0000", 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151123+0000", 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4389999999999999e-06 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151128+0000", 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2449999999999998e-06 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151132+0000", 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9049999999999999e-06 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151198+0000", 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5988000000000006e-05 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151205+0000", 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8770000000000004e-06 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226171+0000", 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074966844000000005 2024-04-20T02:52:58.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226261+0000", 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9975999999999994e-05 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263337+0000", 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075863000000001 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263443+0000", 2024-04-20T02:52:58.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000105678 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263784+0000", 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034103200000000001 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4095 4.9 4:964390c0:::smithi10548823-36:head [write 450520~558231] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.3998420339999997, 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112875926, 2024-04-20T02:52:58.644 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4095 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151134+0000", 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151379+0000", 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024486000000000001 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151385+0000", 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1460000000000001e-06 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151389+0000", 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6610000000000002e-06 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151413+0000", 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3611000000000001e-05 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151419+0000", 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8780000000000003e-06 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226302+0000", 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074883540999999998 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226460+0000", 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015792000000000001 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226599+0000", 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013939599999999999 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262685+0000", 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036086115000000002 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262726+0000", 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0448999999999998e-05 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263833+0000", 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001106685 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263884+0000", 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1530999999999998e-05 2024-04-20T02:52:58.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264011+0000", 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012641900000000001 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4096 4.9 4:964390c0:::smithi10548823-36:head [write 1483047~515125] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.3995824990000001, 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11703358799999999, 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.651 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4096 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:52:58.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151644+0000", 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025029799999999998 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151651+0000", 2024-04-20T02:52:58.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.148e-06 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151655+0000", 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2769999999999999e-06 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151689+0000", 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711999999999998e-05 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151698+0000", 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0359999999999995e-06 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245111+0000", 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093413164000000007 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245232+0000", 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000120214 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:52:58.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245424+0000", 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019272500000000001 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263033+0000", 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017608461999999998 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266386+0000", 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033530980000000001 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268218+0000", 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001832101 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268283+0000", 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5334999999999997e-05 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268428+0000", 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014449999999999999 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4097 4.9 4:964390c0:::smithi10548823-36:head [setxattr _header (57) in=7b,truncate 2050520] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.3993145460000003, 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.117031099, 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4097 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151665+0000", 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.614e-06 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151670+0000", 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151675+0000", 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0560000000000004e-06 2024-04-20T02:52:58.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151738+0000", 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2447999999999996e-05 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151753+0000", 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5007e-05 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264044+0000", 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112291562 2024-04-20T02:52:58.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264165+0000", 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121081 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264421+0000", 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025568599999999998 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266133+0000", 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001711668 2024-04-20T02:52:58.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266755+0000", 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00062224100000000005 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268484+0000", 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001729169 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268545+0000", 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0365000000000002e-05 2024-04-20T02:52:58.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268693+0000", 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014860000000000001 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4098 4.9 4:964390c0:::smithi10548823-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 6.3992970639999998, 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11735205, 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:52:58.664 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4098 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:52:58.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151679+0000", 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151681+0000", 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3140000000000002e-06 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151686+0000", 2024-04-20T02:52:58.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0209999999999999e-06 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151691+0000", 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4700000000000004e-06 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151779+0000", 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8399999999999994e-05 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151788+0000", 2024-04-20T02:52:58.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7129999999999994e-06 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265294+0000", 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11350631899999999 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265391+0000", 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6861000000000007e-05 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268730+0000", 2024-04-20T02:52:58.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003339102 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268841+0000", 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110708 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.269032+0000", 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019057899999999999 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:52:58.669 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:52:58.670 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:58.671 INFO:teuthology.orchestra.run.smithi138.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-04-20T02:52:58.672 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:52:58.672 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.4 2024-04-20T02:52:58.672 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-04-20T02:52:58.672 INFO:tasks.thrashosds.thrasher:applying new sharding: m(1,0-) O(5) L(1,5-) 2024-04-20T02:52:58.672 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-4 --sharding 'm(1,0-) O(5) L(1,5-)' reshard 2024-04-20T02:52:58.673 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:58.672+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:58.677 INFO:tasks.rados.rados.0.smithi105.stdout:1825: expect (ObjNum 327 snap 35 seq_num 327) 2024-04-20T02:52:58.699 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:52:58.772 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:58.772+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:58.809 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:52:58.809 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:52:58.809 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:52:58.809 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:52:58.820 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:52:58.865 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:52:58.872 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:58.873+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:58.884 INFO:tasks.rados.rados.0.smithi105.stdout:1822: expect (ObjNum 360 snap 37 seq_num 360) 2024-04-20T02:52:58.885 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:52:58.905 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:52:58.926 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:52:58.946 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:52:58.966 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:52:58.974 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:52:58.974+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:52:58.986 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:52:58.986 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:52:58.986 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:52:58.986 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:52:58.987 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:52:58.987 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-20T02:52:58.987 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:52:58.987 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:52:58.987 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:52:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:52:58.989 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:52:58.992 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:58.992 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:59.003 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:52:59.025 INFO:tasks.rados.rados.0.smithi105.stdout:1824: expect (ObjNum 355 snap 37 seq_num 355) 2024-04-20T02:52:59.075 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:52:59.075+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:52:59.088 INFO:teuthology.orchestra.run.smithi138.stdout:reshard success 2024-04-20T02:52:59.155 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:59.156 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:59.163 INFO:tasks.rados.rados.0.smithi105.stdout:1828: expect (ObjNum 307 snap 33 seq_num 307) 2024-04-20T02:52:59.166 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:52:59.207 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:59.207+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:59.276 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:52:59.277+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:52:59.296 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:59.296 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:59.307 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:52:59.391 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:59.392+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:59.432 INFO:tasks.rados.rados.0.smithi105.stdout:1826: got (0) Success 2024-04-20T02:52:59.433 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 66 (ObjNum 124 snap 9 seq_num 124) dirty exists 2024-04-20T02:52:59.433 INFO:tasks.rados.rados.0.smithi105.stdout:1823: expect (ObjNum 196 snap 19 seq_num 196) 2024-04-20T02:52:59.478 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:59.478+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:59.479 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:59.479 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:59.490 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:52:59.535 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.4 2024-04-20T02:52:59.536 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-4 fsck 2024-04-20T02:52:59.580 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:52:59.580+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:52:59.609 INFO:tasks.rados.rados.0.smithi105.stdout:1819: finishing write tid 1 to smithi10548823-6 2024-04-20T02:52:59.609 INFO:tasks.rados.rados.0.smithi105.stdout:1821: expect (ObjNum 326 snap 35 seq_num 326) 2024-04-20T02:52:59.668 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:59.668 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:59.678 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:52:59.680 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:59.681+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:59.781 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:59.782+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:59.785 INFO:tasks.rados.rados.0.smithi105.stdout:1819: finishing write tid 2 to smithi10548823-6 2024-04-20T02:52:59.785 INFO:tasks.rados.rados.0.smithi105.stdout:1819: finishing write tid 3 to smithi10548823-6 2024-04-20T02:52:59.785 INFO:tasks.rados.rados.0.smithi105.stdout:1820: done (15 left) 2024-04-20T02:52:59.785 INFO:tasks.rados.rados.0.smithi105.stdout:1821: done (14 left) 2024-04-20T02:52:59.785 INFO:tasks.rados.rados.0.smithi105.stdout:1822: done (13 left) 2024-04-20T02:52:59.785 INFO:tasks.rados.rados.0.smithi105.stdout:1823: done (12 left) 2024-04-20T02:52:59.785 INFO:tasks.rados.rados.0.smithi105.stdout:1824: done (11 left) 2024-04-20T02:52:59.785 INFO:tasks.rados.rados.0.smithi105.stdout:1825: done (10 left) 2024-04-20T02:52:59.785 INFO:tasks.rados.rados.0.smithi105.stdout:1826: done (9 left) 2024-04-20T02:52:59.785 INFO:tasks.rados.rados.0.smithi105.stdout:1828: done (8 left) 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1836: read oid 49 snap -1 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1836: expect (ObjNum 352 snap 37 seq_num 352) 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1837: snap_create 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1819: finishing write tid 4 to smithi10548823-6 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1819: oid 6 updating version 0 to 287 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1819: oid 6 version 287 is already newer than 286 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1819: oid 6 version 287 is already newer than 285 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 287 (ObjNum 371 snap 38 seq_num 371) dirty exists 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1819: left oid 6 (ObjNum 371 snap 38 seq_num 371) 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1827: finishing write tid 1 to smithi10548823-3 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1827: finishing write tid 2 to smithi10548823-3 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1827: finishing write tid 3 to smithi10548823-3 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1827: finishing write tid 4 to smithi10548823-3 2024-04-20T02:52:59.786 INFO:tasks.rados.rados.0.smithi105.stdout:1827: oid 3 updating version 0 to 442 2024-04-20T02:52:59.787 INFO:tasks.rados.rados.0.smithi105.stdout:1827: oid 3 version 442 is already newer than 440 2024-04-20T02:52:59.787 INFO:tasks.rados.rados.0.smithi105.stdout:1827: oid 3 version 442 is already newer than 441 2024-04-20T02:52:59.787 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 3 v 442 (ObjNum 372 snap 38 seq_num 372) dirty exists 2024-04-20T02:52:59.787 INFO:tasks.rados.rados.0.smithi105.stdout:1827: left oid 3 (ObjNum 372 snap 38 seq_num 372) 2024-04-20T02:52:59.787 INFO:tasks.rados.rados.0.smithi105.stdout:1834: got (16) Device or resource busy 2024-04-20T02:52:59.787 INFO:tasks.rados.rados.0.smithi105.stdout:check_oldest_snap_flushed oid 50 v 106 (ObjNum 350 snap 37 seq_num 350) unflushed 38 2024-04-20T02:52:59.787 INFO:tasks.rados.rados.0.smithi105.stdout:1833: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:52:59.806 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:59.806 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:52:59.816 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:52:59.900 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:52:59.901+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:52:59.983 INFO:tasks.rados.rados.0.smithi105.stdout:1831: expect (ObjNum 319 snap 34 seq_num 319) 2024-04-20T02:52:59.983 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:52:59.983+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:52:59.995 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:52:59.995 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:00.011 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:00.083 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:00.083+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:00.169 INFO:tasks.rados.rados.0.smithi105.stdout:1832: expect (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:53:00.183 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:00.184 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:00.184 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:00.184+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:00.194 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:00.285 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:00.285+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:00.323 INFO:tasks.rados.rados.0.smithi105.stdout:1835: got (0) Success 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 136 (ObjNum 336 snap 36 seq_num 336) dirty exists 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:1830: finishing write tid 1 to smithi10548823-23 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:1830: finishing write tid 2 to smithi10548823-23 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:1830: finishing write tid 3 to smithi10548823-23 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:1830: finishing write tid 4 to smithi10548823-23 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:1830: oid 23 updating version 0 to 128 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:1830: oid 23 updating version 128 to 130 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:1830: oid 23 version 130 is already newer than 129 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 23 v 130 (ObjNum 373 snap 38 seq_num 373) dirty exists 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:1830: left oid 23 (ObjNum 373 snap 38 seq_num 373) 2024-04-20T02:53:00.324 INFO:tasks.rados.rados.0.smithi105.stdout:1836: expect (ObjNum 352 snap 37 seq_num 352) 2024-04-20T02:53:00.364 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:00.364 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:00.375 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:00.387 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:00.386+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:00.388 INFO:teuthology.orchestra.run.smithi138.stdout:fsck success 2024-04-20T02:53:00.396 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-04-20T02:53:00.396 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-04-20T02:53:00.396 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-20T02:53:00.396 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-20T02:53:00.430 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:53:00.442 INFO:tasks.ceph.osd.4:Started 2024-04-20T02:53:00.443 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:00.487 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:00.488+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:00.514 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:00.514 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:00.526 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:no valid command found; 10 closest matches: 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:0 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:1 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:2 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:abort 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:assert 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:config diff 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:config diff get 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:config get 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:config help [] 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:config set ... 2024-04-20T02:53:00.572 INFO:teuthology.orchestra.run.smithi138.stderr:admin_socket: invalid command 2024-04-20T02:53:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-20T02:53:00.575 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1819: done (9 left) 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1827: done (8 left) 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1830: done (7 left) 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1831: done (6 left) 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1832: done (5 left) 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1833: done (4 left) 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1834: done (3 left) 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1835: done (2 left) 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1836: done (1 left) 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1837: done (0 left) 2024-04-20T02:53:00.582 INFO:tasks.rados.rados.0.smithi105.stdout:1838: read oid 46 snap -1 2024-04-20T02:53:00.583 INFO:tasks.rados.rados.0.smithi105.stdout:1838: expect (ObjNum 334 snap 36 seq_num 334) 2024-04-20T02:53:00.583 INFO:tasks.rados.rados.0.smithi105.stdout:1839: read oid 50 snap -1 2024-04-20T02:53:00.583 INFO:tasks.rados.rados.0.smithi105.stdout:1839: expect (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:53:00.583 INFO:tasks.rados.rados.0.smithi105.stdout:1840: write oid 33 current snap is 39 2024-04-20T02:53:00.583 INFO:tasks.rados.rados.0.smithi105.stdout:1840: seq_num 374 ranges {685818=644423,2069089=216729} 2024-04-20T02:53:00.588 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:00.588+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:00.589 INFO:tasks.rados.rados.0.smithi105.stdout:1840: writing smithi10548823-33 from 685818 to 1330241 tid 1 2024-04-20T02:53:00.595 INFO:tasks.rados.rados.0.smithi105.stdout:1840: writing smithi10548823-33 from 2069089 to 2285818 tid 2 2024-04-20T02:53:00.595 INFO:tasks.rados.rados.0.smithi105.stdout:1841: read oid 4 snap -1 2024-04-20T02:53:00.595 INFO:tasks.rados.rados.0.smithi105.stdout:1841: expect (ObjNum 351 snap 37 seq_num 351) 2024-04-20T02:53:00.595 INFO:tasks.rados.rados.0.smithi105.stdout:1842: write (excl) oid 42 current snap is 39 2024-04-20T02:53:00.595 INFO:tasks.rados.rados.0.smithi105.stdout:1842: seq_num 375 ranges {702625=719672,2127571=175054} 2024-04-20T02:53:00.602 INFO:tasks.rados.rados.0.smithi105.stdout:1842: writing smithi10548823-42 from 702625 to 1422297 tid 1 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1842: writing smithi10548823-42 from 2127571 to 2302625 tid 2 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1843: read oid 38 snap -1 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1843: expect deleted 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1844: read oid 13 snap -1 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1844: expect (ObjNum 345 snap 37 seq_num 345) 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1845: read oid 43 snap -1 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1845: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1846: read oid 28 snap -1 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1846: expect (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1847: tier_promote oid 7 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1848: read oid 6 snap -1 2024-04-20T02:53:00.609 INFO:tasks.rados.rados.0.smithi105.stdout:1848: expect (ObjNum 371 snap 38 seq_num 371) 2024-04-20T02:53:00.610 INFO:tasks.rados.rados.0.smithi105.stdout:1849: tier_evict oid 23 2024-04-20T02:53:00.610 INFO:tasks.rados.rados.0.smithi105.stdout:1849: tier_evict oid 23 snap -1 2024-04-20T02:53:00.610 INFO:tasks.rados.rados.0.smithi105.stdout:1850: set_chunk oid 12 target oid 2024-04-20T02:53:00.610 INFO:tasks.rados.rados.0.smithi105.stdout:1850: set_chunk oid 12 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:53:00.610 INFO:tasks.rados.rados.0.smithi105.stdout:1851: snap_create 2024-04-20T02:53:00.613 INFO:tasks.rados.rados.0.smithi105.stdout:1841: expect (ObjNum 351 snap 37 seq_num 351) 2024-04-20T02:53:00.668 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:00.668 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:00.678 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:00.723 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:00.723+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:00.790 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:00.790+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:00.808 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:00.808 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:00.819 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:00.858 INFO:tasks.rados.rados.0.smithi105.stdout:1839: expect (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:53:00.891 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:00.891+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:00.988 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:00.989 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:00.990 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:00.992 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:00.992+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:00.999 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:01.033 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:01.033+0000 7f9d41cb77c0 -1 Falling back to public interface 2024-04-20T02:53:01.092 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:01.093+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:01.146 INFO:tasks.rados.rados.0.smithi105.stdout:1838: expect (ObjNum 334 snap 36 seq_num 334) 2024-04-20T02:53:01.152 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:53:01.159 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:01.178 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:01.185 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:01.185 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:01.193 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:01.193+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:01.195 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:01.197 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:01.217 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:01.237 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:01.270 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:01.294 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:01.295+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:01.302 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:01.303 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:01.303 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:01.303 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:01.303 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:01.303 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-20T02:53:01.303 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:01.303 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:01.303 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:53:01.306 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:53:01.327 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:01.327 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:01.327 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:01.327 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:01.338 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:01.375 INFO:tasks.rados.rados.0.smithi105.stdout:1838: done (13 left) 2024-04-20T02:53:01.375 INFO:tasks.rados.rados.0.smithi105.stdout:1839: done (12 left) 2024-04-20T02:53:01.375 INFO:tasks.rados.rados.0.smithi105.stdout:1841: done (11 left) 2024-04-20T02:53:01.375 INFO:tasks.rados.rados.0.smithi105.stdout:1851: done (10 left) 2024-04-20T02:53:01.375 INFO:tasks.rados.rados.0.smithi105.stdout:1852: write oid 48 current snap is 40 2024-04-20T02:53:01.375 INFO:tasks.rados.rados.0.smithi105.stdout:1849: got (22) Invalid argument 2024-04-20T02:53:01.375 INFO:tasks.rados.rados.0.smithi105.stdout:1850: finishing set_chunk to oid 12 2024-04-20T02:53:01.375 INFO:tasks.rados.rados.0.smithi105.stdout:1850: got expected ERANGE (src dne) 2024-04-20T02:53:01.375 INFO:tasks.rados.rados.0.smithi105.stdout:1852: seq_num 376 ranges {719432=794921,2186053=133379} 2024-04-20T02:53:01.382 INFO:tasks.rados.rados.0.smithi105.stdout:1852: writing smithi10548823-48 from 719432 to 1514353 tid 1 2024-04-20T02:53:01.388 INFO:tasks.rados.rados.0.smithi105.stdout:1852: writing smithi10548823-48 from 2186053 to 2319432 tid 2 2024-04-20T02:53:01.388 INFO:tasks.rados.rados.0.smithi105.stdout:1843: done (10 left) 2024-04-20T02:53:01.388 INFO:tasks.rados.rados.0.smithi105.stdout:1849: done (9 left) 2024-04-20T02:53:01.388 INFO:tasks.rados.rados.0.smithi105.stdout:1850: done (8 left) 2024-04-20T02:53:01.388 INFO:tasks.rados.rados.0.smithi105.stdout:1853: read oid 32 snap -1 2024-04-20T02:53:01.388 INFO:tasks.rados.rados.0.smithi105.stdout:1853: expect (ObjNum 342 snap 37 seq_num 342) 2024-04-20T02:53:01.388 INFO:tasks.rados.rados.0.smithi105.stdout:1854: read oid 15 snap -1 2024-04-20T02:53:01.388 INFO:tasks.rados.rados.0.smithi105.stdout:1854: expect deleted 2024-04-20T02:53:01.388 INFO:tasks.rados.rados.0.smithi105.stdout:1855: snap_remove snap 35 2024-04-20T02:53:01.395 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:01.396+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:01.497 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:01.498+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:01.508 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:01.508 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:01.508 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:01.508 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:01.509 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:01.518 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:01.603 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:01.603+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:01.689 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:01.689 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:01.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:01.689 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:01.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:968 4.0 e32/22 4:007b2a87:::smithi10548823-39:head v 32'32, mlcod=30'30)", 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 135.14370214799999, 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062396513000000001, 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544660+0000", 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544659+0000", 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544944+0000", 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028520200000000002 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544961+0000", 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7544e-05 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544994+0000", 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2681000000000001e-05 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545066+0000", 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2092e-05 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.545102+0000", 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5661000000000003e-05 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606996+0000", 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061894149000000002 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607057+0000", 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0523000000000003e-05 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 130.755779687, 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.694 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:53:01.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 130.75551948099999, 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:53:01.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:53:01.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:53:01.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:53:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:53:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:53:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:53:01.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:53:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:53:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:53:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 130.75549801299999, 2024-04-20T02:53:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:53:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:01.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:01.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:53:01.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:01.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(osd.4.15:128 3.1 3:93d296fd:::c0bd8d7f9abe08d6c342a7099694dcbf39f491ac1a2e8596fc2c5deb03fa265e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 121.56182608100001, 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061533704000000002, 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126536+0000", 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.126535+0000", 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:01.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127218+0000", 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00068305299999999998 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127245+0000", 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7129000000000001e-05 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.127253+0000", 2024-04-20T02:53:01.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4910000000000005e-06 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128249+0000", 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00099557899999999991 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128392+0000", 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014281400000000001 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.128616+0000", 2024-04-20T02:53:01.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000223909 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.141935+0000", 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013319609 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.155334+0000", 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013398554999999999 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188003+0000", 2024-04-20T02:53:01.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032669198000000003 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188046+0000", 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3350999999999999e-05 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:00.188070+0000", 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3485999999999998e-05 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2292 4.3 4:c07c6daa:::smithi10548823-7:head [write 544593~799729] snapc 1e=[1e,1d,1b,18,16,15,13,d] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 87.528782888999999, 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061161747000000002, 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2292 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159579+0000", 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.159577+0000", 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999971 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160019+0000", 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000442674 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160045+0000", 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5446e-05 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160058+0000", 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3444e-05 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160487+0000", 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000428899 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160616+0000", 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012861099999999999 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.160838+0000", 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000222415 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207115+0000", 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.046276543000000003 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.207164+0000", 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9497999999999999e-05 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220657+0000", 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013492386 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220698+0000", 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1451e-05 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:34.220741+0000", 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3078000000000003e-05 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:2699 4.0 e95/60 4:007b2a87:::smithi10548823-39:head v 95'100, mlcod=94'97)", 2024-04-20T02:53:01.723 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.339201600999999, 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061625529999999998, 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349161+0000", 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349160+0000", 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349351+0000", 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019061600000000001 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349364+0000", 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2835e-05 2024-04-20T02:53:01.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349376+0000", 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2422999999999999e-05 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349466+0000", 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9615000000000002e-05 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349492+0000", 2024-04-20T02:53:01.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6494000000000001e-05 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.410234+0000", 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.060741334000000001 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.410786+0000", 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000552539 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3423 4.3 4:c07c6daa:::smithi10548823-7:head [write 558065~619869] snapc 3a=[3a,39,38,36,35,33] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 33.298913865999999, 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061409060000000001, 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3423 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389448+0000", 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389446+0000", 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389586+0000", 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014017899999999999 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389605+0000", 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.389617+0000", 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1956999999999999e-05 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.412832+0000", 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.023215612 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.412909+0000", 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6555000000000001e-05 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.413035+0000", 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000126599 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450700+0000", 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037664717 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450712+0000", 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1422e-05 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450741+0000", 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9791999999999999e-05 2024-04-20T02:53:01.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450801+0000", 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9199e-05 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:28.450857+0000", 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6882999999999997e-05 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.733 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3771 4.b 4:d4308fef:::smithi10548823-6:head [write 2548663~763022] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.506469356, 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072733780999999997, 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.734 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3771 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181891+0000", 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182301+0000", 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040985099999999998 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182317+0000", 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6305999999999999e-05 2024-04-20T02:53:01.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182329+0000", 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1337e-05 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198720+0000", 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016391606 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198807+0000", 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7319999999999997e-05 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198925+0000", 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117345 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218062+0000", 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019137134 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234990+0000", 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016928367 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254488+0000", 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019497460000000001 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254571+0000", 2024-04-20T02:53:01.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3029999999999996e-05 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254627+0000", 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5865000000000001e-05 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3772 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3395047] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.506022783999999, 2024-04-20T02:53:01.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072419708999999999, 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3772 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182343+0000", 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0099999999999997e-06 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182349+0000", 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9569999999999999e-06 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182354+0000", 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.826e-06 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217228+0000", 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034874555000000002 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217312+0000", 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3478000000000003e-05 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217431+0000", 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119512 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234969+0000", 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017537833999999999 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235012+0000", 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3411000000000002e-05 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254638+0000", 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019625815000000001 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254682+0000", 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3945e-05 2024-04-20T02:53:01.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254759+0000", 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6882e-05 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3773 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.506004177000001, 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072538680999999994, 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.746 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3773 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182360+0000", 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3309999999999998e-06 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182366+0000", 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.981e-06 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182371+0000", 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5390000000000001e-06 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217932+0000", 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035561700000000002 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:01.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218020+0000", 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7352e-05 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254778+0000", 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036757971 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254819+0000", 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1525e-05 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254897+0000", 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7621999999999996e-05 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3774 4.3 4:c07c6daa:::smithi10548823-7:head [write 611854~717004] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.190321+0000", 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.498041023999999, 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064562913999999999, 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.751 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3774 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190321+0000", 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190321+0000", 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190319+0000", 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190704+0000", 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038458399999999999 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190721+0000", 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6988000000000001e-05 2024-04-20T02:53:01.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.190734+0000", 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.276e-05 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217683+0000", 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026949832 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217775+0000", 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1669000000000006e-05 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217991+0000", 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021635400000000001 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254534+0000", 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036542842999999998 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254573+0000", 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8921000000000002e-05 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254780+0000", 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020691700000000001 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254833+0000", 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2785999999999999e-05 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254884+0000", 2024-04-20T02:53:01.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1159999999999998e-05 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3775 4.3 4:c07c6daa:::smithi10548823-7:head [write 2126143~561767] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.198678+0000", 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.489683763999999, 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062905904999999998, 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.757 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3775 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198678+0000", 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198678+0000", 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198677+0000", 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198991+0000", 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00031458700000000002 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.199005+0000", 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3134e-05 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.199013+0000", 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3489999999999992e-06 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234979+0000", 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035965991000000003 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235102+0000", 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012322400000000001 2024-04-20T02:53:01.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235329+0000", 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022682799999999999 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254591+0000", 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019262029999999999 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254607+0000", 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6150999999999998e-05 2024-04-20T02:53:01.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.261494+0000", 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068870779999999996 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.261539+0000", 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4400000000000002e-05 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.261584+0000", 2024-04-20T02:53:01.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5775000000000001e-05 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3880 4.3 4:c32272c0:::smithi10548823-41:head [write 796731~461096] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.406265+0000", 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.282097259, 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066110050000000004, 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.763 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3880 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406265+0000", 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406265+0000", 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406264+0000", 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406481+0000", 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021666200000000001 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406493+0000", 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1977999999999999e-05 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406499+0000", 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3609999999999997e-06 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406540+0000", 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0728e-05 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406632+0000", 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1821000000000001e-05 2024-04-20T02:53:01.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.406742+0000", 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110178 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.459209+0000", 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.052467645 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.459342+0000", 2024-04-20T02:53:01.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013213 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.472265+0000", 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012923128000000001 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.472321+0000", 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.63e-05 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.472375+0000", 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4042999999999997e-05 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4057 4.b 4:d4308fef:::smithi10548823-6:head [write 749678~790384,stat] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.5451494369999992, 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074563737000000005, 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.769 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4057 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143210+0000", 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143760+0000", 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055015999999999995 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143772+0000", 2024-04-20T02:53:01.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1345000000000001e-05 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143781+0000", 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0929999999999992e-06 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143895+0000", 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113998 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:01.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143906+0000", 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1195999999999999e-05 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.153590+0000", 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096836000000000005 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155611+0000", 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002021012 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217691+0000", 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062080244 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217777+0000", 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5583999999999999e-05 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4058 4.b 4:d4308fef:::smithi10548823-6:head [write 2332155~504118] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:01.774 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.5445684110000002, 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074090323999999999, 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4058 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:01.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143793+0000", 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144104+0000", 2024-04-20T02:53:01.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000310971 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144112+0000", 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1990000000000006e-06 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144118+0000", 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.443e-06 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144162+0000", 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3066999999999999e-05 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144172+0000", 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0083e-05 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155911+0000", 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011739657000000001 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155993+0000", 2024-04-20T02:53:01.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1754000000000004e-05 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217810+0000", 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061816958999999998 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217884+0000", 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4098999999999996e-05 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4059 4.b 4:d4308fef:::smithi10548823-6:head [write 3271857~549469] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.5442322500000003, 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073860173000000001, 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4059 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144129+0000", 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144439+0000", 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030924000000000001 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144447+0000", 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8930000000000005e-06 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144455+0000", 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1410000000000005e-06 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144505+0000", 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0698000000000001e-05 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144520+0000", 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4605e-05 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156034+0000", 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011513809 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156130+0000", 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6039999999999995e-05 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217915+0000", 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061785501999999999 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217990+0000", 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4770000000000004e-05 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.784 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4060 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3949678] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.5438968709999994, 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085188106999999999, 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4060 2024-04-20T02:53:01.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144469+0000", 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.946e-06 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144476+0000", 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7909999999999999e-06 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144482+0000", 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.601e-06 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144803+0000", 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032103699999999998 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144813+0000", 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0183999999999999e-05 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156168+0000", 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011354523 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156254+0000", 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5706999999999998e-05 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218023+0000", 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061769566999999997 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218137+0000", 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113912 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218377+0000", 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023941000000000001 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225107+0000", 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067304510000000001 2024-04-20T02:53:01.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225121+0000", 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4341e-05 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229481+0000", 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043599880000000004 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229522+0000", 2024-04-20T02:53:01.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0224e-05 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229653+0000", 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013190500000000001 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4061 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.5437572539999991, 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085431572999999997, 2024-04-20T02:53:01.792 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4061 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144604+0000", 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144608+0000", 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1080000000000003e-06 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144618+0000", 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5149999999999995e-06 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144625+0000", 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8260000000000004e-06 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144846+0000", 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022160699999999999 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144854+0000", 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0399999999999993e-06 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156327+0000", 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011472178 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156457+0000", 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013012599999999999 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218704+0000", 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062247080000000003 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218795+0000", 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1391000000000006e-05 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229703+0000", 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010907834999999999 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229818+0000", 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115423 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.230037+0000", 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021803600000000001 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4146 3.1 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head [write 0~1129472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:01.798 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.5081612769999992, 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067860845000000003, 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4146 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:01.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180199+0000", 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180439+0000", 2024-04-20T02:53:01.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000239375 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180449+0000", 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0220999999999999e-05 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180456+0000", 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0090000000000001e-06 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:01.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181170+0000", 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071358900000000004 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181331+0000", 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016178500000000001 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,5", 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181490+0000", 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158703 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225260+0000", 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043770166999999999 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226398+0000", 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001138189 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247971+0000", 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021572728999999999 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248021+0000", 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0176999999999997e-05 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248062+0000", 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0472000000000001e-05 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:01.804 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:01.806 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:01.806 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:01.806+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:01.907 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:01.907+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:01.973 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:01.974 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:01.974 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:01.974 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:01.984 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:02.028 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:02.028+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:02.108 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:02.109+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:02.113 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:02.113 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:02.113 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:02.113 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:02.123 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:02.158 INFO:tasks.rados.rados.0.smithi105.stdout:1855: done (10 left) 2024-04-20T02:53:02.158 INFO:tasks.rados.rados.0.smithi105.stdout:1856: write oid 34 current snap is 40 2024-04-20T02:53:02.158 INFO:tasks.rados.rados.0.smithi105.stdout:1845: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:53:02.209 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:02.210+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:02.300 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:02.300 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:02.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:02.300 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 135.71394852099999, 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.301 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:53:02.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:53:02.307 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 135.69321921, 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.308 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:02.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:53:02.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:53:02.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 135.683967459, 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.315 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:53:02.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:53:02.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 127.886968243, 2024-04-20T02:53:02.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:53:02.323 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.323 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.323 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:02.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:53:02.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 127.880963455, 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:53:02.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.332 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4029 4.13 4:cf11ccc0:::smithi10548823-4:head [write 699257~564637,stat] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.743420939, 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.86467550400000004, 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.333 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4029 2024-04-20T02:53:02.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.334 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:02.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:02.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555666+0000", 2024-04-20T02:53:02.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555809+0000", 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014306900000000001 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555832+0000", 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3680999999999999e-05 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.336 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555846+0000", 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3654e-05 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555913+0000", 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6736999999999995e-05 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556058+0000", 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014538100000000001 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:02.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556392+0000", 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033393099999999998 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578661+0000", 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022268553999999999 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578702+0000", 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1533999999999999e-05 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578747+0000", 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4858999999999999e-05 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578811+0000", 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3520000000000002e-05 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605743+0000", 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026932418999999999 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420312+0000", 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81456849600000003 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420343+0000", 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1911e-05 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.340 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4030 4.13 4:cf11ccc0:::smithi10548823-4:head [write 1873062~640438] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.736507818, 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85780235299999996, 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4030 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:02.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562579+0000", 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.342 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562714+0000", 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013475 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562728+0000", 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3516000000000001e-05 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562736+0000", 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5690000000000005e-06 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578835+0000", 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016099156 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578957+0000", 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121094 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.579123+0000", 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016647100000000001 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605727+0000", 2024-04-20T02:53:02.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026603643999999999 2024-04-20T02:53:02.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605749+0000", 2024-04-20T02:53:02.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2070999999999999e-05 2024-04-20T02:53:02.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605799+0000", 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0417000000000002e-05 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605841+0000", 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2344e-05 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605907+0000", 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5135999999999994e-05 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420373+0000", 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81446675899999998 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420383+0000", 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0108999999999999e-05 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4031 4.13 4:cf11ccc0:::smithi10548823-4:head [write 3305941~593316] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.728946242999999, 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85027709799999995, 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4031 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.348 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570142+0000", 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570277+0000", 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135606 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570288+0000", 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0719e-05 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570294+0000", 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2519999999999996e-06 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592786+0000", 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022491220999999999 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592911+0000", 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012546599999999999 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.593118+0000", 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000207011 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605878+0000", 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012760298999999999 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605934+0000", 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5813999999999998e-05 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605963+0000", 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9275e-05 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606612+0000", 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00064813999999999996 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613782+0000", 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071705670000000001 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420411+0000", 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.80662907399999995 2024-04-20T02:53:02.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420420+0000", 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5790000000000004e-06 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4032 4.13 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (56) in=7b,truncate 3899257] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.728789815000001, 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85015831100000006, 2024-04-20T02:53:02.354 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4032 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570302+0000", 2024-04-20T02:53:02.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1700000000000001e-06 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570305+0000", 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.976e-06 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570308+0000", 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8899999999999999e-06 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605985+0000", 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035677121999999999 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606066+0000", 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0797000000000006e-05 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606211+0000", 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014569999999999999 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607013+0000", 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00080160599999999995 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607046+0000", 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2938999999999998e-05 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607086+0000", 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0203000000000001e-05 2024-04-20T02:53:02.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607157+0000", 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0621e-05 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613833+0000", 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066762269999999999 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420448+0000", 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.806615478 2024-04-20T02:53:02.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420457+0000", 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9779999999999994e-06 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4033 4.13 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.728777941000001, 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85021192700000003, 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.361 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4033 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570312+0000", 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419999999999999e-06 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570316+0000", 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3730000000000001e-06 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570318+0000", 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6680000000000001e-06 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606541+0000", 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036222797000000001 2024-04-20T02:53:02.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606598+0000", 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6629999999999998e-05 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420515+0000", 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81391740099999998 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420523+0000", 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6799999999999993e-06 2024-04-20T02:53:02.365 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4034 4.13 4:cfd1e4d4:::smithi10548823-9:head [rollback 3f] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.728768303000001, 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85017147900000001, 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.366 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4034 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570320+0000", 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570322+0000", 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.37e-06 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570325+0000", 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7800000000000001e-06 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570328+0000", 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5379999999999998e-06 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606632+0000", 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036304075999999998 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606716+0000", 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3689000000000005e-05 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420484+0000", 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81376812700000001 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420492+0000", 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1179999999999992e-06 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4062 4.9 4:9c87d36d:::smithi10548823-24:head [write 766485~465633] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:02.370 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.148914522, 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075029242999999995, 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4062 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150173+0000", 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150435+0000", 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000262064 2024-04-20T02:53:02.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150445+0000", 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8160000000000005e-06 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150450+0000", 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1880000000000001e-06 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150484+0000", 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:02.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150490+0000", 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6810000000000003e-06 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161686+0000", 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011196187 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161840+0000", 2024-04-20T02:53:02.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015389000000000001 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.162103+0000", 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026333299999999998 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224927+0000", 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062823458999999998 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224993+0000", 2024-04-20T02:53:02.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6667000000000004e-05 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225059+0000", 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6117000000000004e-05 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225119+0000", 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9342000000000002e-05 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225204+0000", 2024-04-20T02:53:02.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4844000000000003e-05 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4063 4.9 4:9c87d36d:::smithi10548823-24:head [write 1990637~647776] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.148631848000001, 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075097435000000004, 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.377 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4063 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150746+0000", 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028951399999999999 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150753+0000", 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150757+0000", 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2300000000000002e-06 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150801+0000", 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3421000000000003e-05 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150811+0000", 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0581000000000001e-05 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177065+0000", 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02625367 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177164+0000", 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8900999999999994e-05 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177319+0000", 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000155437 2024-04-20T02:53:02.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225048+0000", 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047728857 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225217+0000", 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168395 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225375+0000", 2024-04-20T02:53:02.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157837 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225437+0000", 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2824999999999996e-05 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225555+0000", 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117169 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4064 4.9 4:9c87d36d:::smithi10548823-24:head [write 3299280~667205] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.148325095000001, 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112222372, 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4064 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150763+0000", 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151076+0000", 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000312355 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151083+0000", 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1010000000000003e-06 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151091+0000", 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0409999999999998e-06 2024-04-20T02:53:02.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151127+0000", 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151133+0000", 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1099999999999999e-06 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200197+0000", 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049064000000000003 2024-04-20T02:53:02.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200315+0000", 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117855 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200545+0000", 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022990700000000001 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225595+0000", 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025050224999999999 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262670+0000", 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075460999999997 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262767+0000", 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6479999999999998e-05 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262835+0000", 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999999e-05 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262986+0000", 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015129799999999999 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4065 4.9 4:9c87d36d:::smithi10548823-24:head [setxattr _header (57) in=7b,truncate 3966485] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.390 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.147985215, 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112193353, 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4065 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151103+0000", 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151107+0000", 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:02.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151111+0000", 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2309999999999998e-06 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151116+0000", 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4850000000000003e-06 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151168+0000", 2024-04-20T02:53:02.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2673000000000002e-05 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151179+0000", 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0112000000000001e-05 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225611+0000", 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074432549000000001 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225689+0000", 2024-04-20T02:53:02.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7612000000000002e-05 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225858+0000", 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016945400000000001 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262650+0000", 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036792126000000001 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262678+0000", 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8132000000000001e-05 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263077+0000", 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039868299999999999 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263141+0000", 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3448999999999995e-05 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263297+0000", 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015649699999999999 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4066 4.9 4:9c87d36d:::smithi10548823-24:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.147968359, 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112663448, 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4066 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151120+0000", 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151123+0000", 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4389999999999999e-06 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151128+0000", 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2449999999999998e-06 2024-04-20T02:53:02.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151132+0000", 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9049999999999999e-06 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151198+0000", 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5988000000000006e-05 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151205+0000", 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8770000000000004e-06 2024-04-20T02:53:02.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226171+0000", 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074966844000000005 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226261+0000", 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9975999999999994e-05 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263337+0000", 2024-04-20T02:53:02.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075863000000001 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263443+0000", 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000105678 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263784+0000", 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034103200000000001 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4095 4.9 4:964390c0:::smithi10548823-36:head [write 450520~558231] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.147954288999999, 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112875926, 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4095 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151134+0000", 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151379+0000", 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024486000000000001 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151385+0000", 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1460000000000001e-06 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151389+0000", 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6610000000000002e-06 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151413+0000", 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3611000000000001e-05 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151419+0000", 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8780000000000003e-06 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226302+0000", 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074883540999999998 2024-04-20T02:53:02.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226460+0000", 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015792000000000001 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226599+0000", 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013939599999999999 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262685+0000", 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036086115000000002 2024-04-20T02:53:02.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262726+0000", 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0448999999999998e-05 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263833+0000", 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001106685 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263884+0000", 2024-04-20T02:53:02.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1530999999999998e-05 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264011+0000", 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012641900000000001 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4096 4.9 4:964390c0:::smithi10548823-36:head [write 1483047~515125] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.147694754, 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11703358799999999, 2024-04-20T02:53:02.409 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4096 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151644+0000", 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025029799999999998 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151651+0000", 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.148e-06 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151655+0000", 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2769999999999999e-06 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151689+0000", 2024-04-20T02:53:02.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711999999999998e-05 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151698+0000", 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0359999999999995e-06 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245111+0000", 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093413164000000007 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245232+0000", 2024-04-20T02:53:02.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000120214 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245424+0000", 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019272500000000001 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263033+0000", 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017608461999999998 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266386+0000", 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033530980000000001 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268218+0000", 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001832101 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268283+0000", 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5334999999999997e-05 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268428+0000", 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014449999999999999 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4097 4.9 4:964390c0:::smithi10548823-36:head [setxattr _header (57) in=7b,truncate 2050520] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.147426801, 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.117031099, 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.416 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4097 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:02.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151665+0000", 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.614e-06 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151670+0000", 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151675+0000", 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0560000000000004e-06 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151738+0000", 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2447999999999996e-05 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151753+0000", 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5007e-05 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264044+0000", 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112291562 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264165+0000", 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121081 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264421+0000", 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025568599999999998 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266133+0000", 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001711668 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266755+0000", 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00062224100000000005 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268484+0000", 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001729169 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268545+0000", 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0365000000000002e-05 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268693+0000", 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014860000000000001 2024-04-20T02:53:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4098 4.9 4:964390c0:::smithi10548823-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.147409318999999, 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11735205, 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4098 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151679+0000", 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151681+0000", 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3140000000000002e-06 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151686+0000", 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0209999999999999e-06 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151691+0000", 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4700000000000004e-06 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151779+0000", 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8399999999999994e-05 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151788+0000", 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7129999999999994e-06 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265294+0000", 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11350631899999999 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265391+0000", 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6861000000000007e-05 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268730+0000", 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003339102 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268841+0000", 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110708 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.269032+0000", 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019057899999999999 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.428 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:02.430 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:02.430 INFO:tasks.rados.rados.0.smithi105.stdout:1847: got (0) Success 2024-04-20T02:53:02.430 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 242 (ObjNum 359 snap 37 seq_num 359) dirty exists 2024-04-20T02:53:02.430 INFO:tasks.rados.rados.0.smithi105.stdout:1848: expect (ObjNum 371 snap 38 seq_num 371) 2024-04-20T02:53:02.431 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:02.430+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:02.494 INFO:tasks.rados.rados.0.smithi105.stdout:1840: finishing write tid 1 to smithi10548823-33 2024-04-20T02:53:02.494 INFO:tasks.rados.rados.0.smithi105.stdout:1846: expect (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:53:02.530 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:02.531+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:02.560 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:02.561 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:02.561 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:02.561 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:02.571 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:02.631 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:02.632+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:02.700 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:02.700 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:02.701 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:02.701 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:02.712 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:02.719 INFO:tasks.rados.rados.0.smithi105.stdout:1844: expect (ObjNum 345 snap 37 seq_num 345) 2024-04-20T02:53:02.763 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:02.763+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:02.833 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:02.833+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 136.30701052000001, 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.852 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:53:02.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:53:02.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:53:02.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:02.857 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 128.47846163200001, 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.858 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.859 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:53:02.860 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:53:02.861 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.862 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.863 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 128.46220944999999, 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.864 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.865 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.866 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.867 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 70.502510431000005, 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:53:02.868 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:53:02.869 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:53:02.870 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:53:02.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:53:02.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.873 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3872 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 779924~785847,stat] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.500107192000002, 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070878165000000007, 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3872 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350641+0000", 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351179+0000", 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053797199999999995 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351196+0000", 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.66e-05 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351209+0000", 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2952000000000001e-05 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351306+0000", 2024-04-20T02:53:02.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7272e-05 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351322+0000", 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6416e-05 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358164+0000", 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068416019999999996 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358336+0000", 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017156499999999999 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358594+0000", 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025794499999999998 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389148+0000", 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030554773 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409688+0000", 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020539497 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420893+0000", 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011205503 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420934+0000", 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1171000000000003e-05 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420998+0000", 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3386999999999995e-05 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421521+0000", 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052313200000000005 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3873 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 2078257~580112] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.499523906, 2024-04-20T02:53:02.880 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075737411000000004, 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3873 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351225+0000", 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351614+0000", 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038825099999999999 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351625+0000", 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1173e-05 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351634+0000", 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0489999999999993e-06 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351696+0000", 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1885999999999997e-05 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351708+0000", 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2702e-05 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389193+0000", 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037484574999999999 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389303+0000", 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110188 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389563+0000", 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026034499999999999 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420846+0000", 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031283014999999997 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421592+0000", 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074529299999999995 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421632+0000", 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0677999999999999e-05 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426683+0000", 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050507119999999997 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426752+0000", 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8484000000000004e-05 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426964+0000", 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021193500000000001 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3874 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 3301257~278667] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.499088905000001, 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075720365999999997, 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.887 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3874 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351660+0000", 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351852+0000", 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019116900000000001 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.889 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351861+0000", 2024-04-20T02:53:02.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6050000000000004e-06 2024-04-20T02:53:02.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351869+0000", 2024-04-20T02:53:02.890 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4780000000000002e-06 2024-04-20T02:53:02.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.890 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.890 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.890 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351916+0000", 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7222999999999998e-05 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351927+0000", 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0791000000000001e-05 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409721+0000", 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057793887000000002 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409840+0000", 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119837 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.410063+0000", 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022257200000000001 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421678+0000", 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011615448 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421721+0000", 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2191000000000003e-05 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.423989+0000", 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022682879999999998 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427051+0000", 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003061785 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427157+0000", 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106684 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427381+0000", 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022416299999999999 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3875 4.10 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (57) in=7b,truncate 3579924] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:02.894 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.498867744000002, 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075851731000000006, 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3875 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351886+0000", 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.561e-06 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351893+0000", 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6100000000000002e-06 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351899+0000", 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8660000000000002e-06 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352002+0000", 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103811 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352012+0000", 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4110000000000002e-06 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421749+0000", 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.069736690000000004 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421861+0000", 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112587 2024-04-20T02:53:02.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422120+0000", 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025878099999999997 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424547+0000", 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024270379999999999 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424960+0000", 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041259899999999998 2024-04-20T02:53:02.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426257+0000", 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001297231 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427489+0000", 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012320949999999999 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427562+0000", 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2825999999999997e-05 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427734+0000", 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017230699999999999 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3876 4.10 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.498845838000001, 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076230629999999994, 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:02.901 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3876 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:02.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351906+0000", 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6350000000000002e-06 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351912+0000", 2024-04-20T02:53:02.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8209999999999998e-06 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351918+0000", 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3839999999999996e-06 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352035+0000", 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117207 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352043+0000", 2024-04-20T02:53:02.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0169999999999996e-06 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422511+0000", 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070467815000000003 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422619+0000", 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010819600000000001 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427772+0000", 2024-04-20T02:53:02.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005153398 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427900+0000", 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012740399999999999 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.428135+0000", 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235138 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4067 4.c 4:392c1e2f:::smithi10548823-44:head [write 783292~540882] snapc 47=[47,45,3f] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.708153629, 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072910900000000001, 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4067 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142595+0000", 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142941+0000", 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034585799999999999 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142954+0000", 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3491e-05 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142965+0000", 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1139e-05 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143003+0000", 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7499000000000001e-05 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143011+0000", 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5490000000000006e-06 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156033+0000", 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013021576999999999 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156242+0000", 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020885500000000001 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156474+0000", 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195900000000001 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177835+0000", 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021361594000000001 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178072+0000", 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000237304 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.214963+0000", 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036890158999999999 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215046+0000", 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3399000000000006e-05 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215507+0000", 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046127700000000001 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4068 4.c 4:392c1e2f:::smithi10548823-44:head [write 1932766~507787] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.707774686, 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073156952999999997, 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.913 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4068 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143300+0000", 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000325225 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143308+0000", 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3860000000000007e-06 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143315+0000", 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7000000000000002e-06 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143360+0000", 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5475000000000001e-05 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143373+0000", 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2204e-05 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178109+0000", 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034736547999999999 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178222+0000", 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112959 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178452+0000", 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000229686 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215864+0000", 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037412038000000002 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215878+0000", 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4029000000000001e-05 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215929+0000", 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0738000000000001e-05 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215989+0000", 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0350999999999999e-05 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216132+0000", 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014332099999999999 2024-04-20T02:53:02.919 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4069 4.c 4:392c1e2f:::smithi10548823-44:head [write 3210350~572013] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.707425873, 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108473219, 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4069 2024-04-20T02:53:02.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143664+0000", 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034090500000000002 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143673+0000", 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1119999999999996e-06 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143679+0000", 2024-04-20T02:53:02.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3589999999999996e-06 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143724+0000", 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4974e-05 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143734+0000", 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0275000000000001e-05 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197473+0000", 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053738971000000003 2024-04-20T02:53:02.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197616+0000", 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014275900000000001 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197834+0000", 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021806000000000001 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251517+0000", 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053683445000000003 2024-04-20T02:53:02.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251579+0000", 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1255999999999998e-05 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251590+0000", 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1399e-05 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251657+0000", 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7039999999999995e-05 2024-04-20T02:53:02.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251797+0000", 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000140235 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4070 4.c 4:392c1e2f:::smithi10548823-44:head [setxattr _header (57) in=7b,truncate 3983292] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.707063085, 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108344988, 2024-04-20T02:53:02.926 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4070 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143686+0000", 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143690+0000", 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7579999999999998e-06 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143697+0000", 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0119999999999999e-06 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143702+0000", 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.78e-06 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143758+0000", 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640999999999998e-05 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143766+0000", 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9519999999999994e-06 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216172+0000", 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072406561999999994 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216295+0000", 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122452 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216531+0000", 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023635999999999999 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251815+0000", 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035283760999999997 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251856+0000", 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1270999999999998e-05 2024-04-20T02:53:02.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251892+0000", 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251929+0000", 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7055999999999999e-05 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252032+0000", 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103183 2024-04-20T02:53:02.932 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4071 4.c 4:392c1e2f:::smithi10548823-44:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.707027886000001, 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108670215, 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.933 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4071 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143721+0000", 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143727+0000", 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2510000000000002e-06 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143739+0000", 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2649e-05 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143749+0000", 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3340000000000008e-06 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143788+0000", 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9773e-05 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143796+0000", 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8720000000000002e-06 2024-04-20T02:53:02.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216846+0000", 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073049823 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216948+0000", 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101525 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252070+0000", 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035122911999999999 2024-04-20T02:53:02.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252180+0000", 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010907499999999999 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252392+0000", 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021285400000000001 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.938 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4106 4.c 4:3d524298:::smithi10548823-18:head [write 484134~708729] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.706994781000001, 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108955254, 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4106 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:02.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144188+0000", 2024-04-20T02:53:02.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000433771 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144198+0000", 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4970000000000007e-06 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144205+0000", 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3409999999999997e-06 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145714+0000", 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015088110000000001 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145727+0000", 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2989e-05 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217011+0000", 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071283814000000001 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217168+0000", 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015707300000000001 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217397+0000", 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022881900000000001 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251870+0000", 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034473539999999997 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252459+0000", 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000589151 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252511+0000", 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1282999999999997e-05 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252569+0000", 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8294999999999997e-05 2024-04-20T02:53:02.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252711+0000", 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000141435 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4107 4.c 4:3d524298:::smithi10548823-18:head [write 1883658~200476] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.706532716, 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.10880477099999999, 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.945 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4107 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144216+0000", 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144346+0000", 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000129732 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144355+0000", 2024-04-20T02:53:02.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4300000000000006e-06 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144361+0000", 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9150000000000001e-06 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145754+0000", 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0013929649999999999 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145763+0000", 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2990000000000002e-06 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243874+0000", 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.098111462999999996 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243990+0000", 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0001156 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:02.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.244226+0000", 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235839 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252760+0000", 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085345439999999998 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252817+0000", 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6842000000000002e-05 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252829+0000", 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1817999999999999e-05 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252884+0000", 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4610000000000001e-05 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253022+0000", 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013853800000000001 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.951 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4108 4.c 4:3d524298:::smithi10548823-18:head [setxattr _header (57) in=7b,truncate 2084134] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.706384291999999, 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.120324168, 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4108 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144365+0000", 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144369+0000", 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8500000000000004e-06 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144374+0000", 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0080000000000002e-06 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144379+0000", 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8239999999999999e-06 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145790+0000", 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014115449999999999 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145800+0000", 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0429999999999996e-06 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253101+0000", 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.107301173 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253210+0000", 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109146 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253456+0000", 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024599700000000002 2024-04-20T02:53:02.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254623+0000", 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001166718 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.258220+0000", 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003597135 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264486+0000", 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062665819999999997 2024-04-20T02:53:02.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264558+0000", 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1364000000000003e-05 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264690+0000", 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013227099999999999 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4109 4.c 4:3d524298:::smithi10548823-18:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.706366770000001, 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12051582399999999, 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4109 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144385+0000", 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2869999999999999e-06 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144392+0000", 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5749999999999997e-06 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144397+0000", 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3589999999999999e-06 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145829+0000", 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00143191 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145838+0000", 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254052+0000", 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108214029 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:02.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254156+0000", 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010343799999999999 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264726+0000", 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010570144 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264796+0000", 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9886000000000001e-05 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264899+0000", 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103336 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:4146 3.1 e153/152 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head v 153'228, mlcod=0'0)", 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 10.668970584, 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066047175, 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181778+0000", 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181997+0000", 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021896 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182011+0000", 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3339999999999999e-05 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182021+0000", 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.046e-05 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182068+0000", 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6931999999999997e-05 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182104+0000", 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6321e-05 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247786+0000", 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065681787000000005 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247827+0000", 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0491000000000002e-05 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4183 3.2 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head [write 0~2392064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.1993364119999992, 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064859071000000004, 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:02.968 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4183 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651396+0000", 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999819 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652285+0000", 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088949500000000002 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652306+0000", 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0812e-05 2024-04-20T02:53:02.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652333+0000", 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7086000000000001e-05 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652373+0000", 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9690000000000001e-05 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652499+0000", 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012642299999999999 2024-04-20T02:53:02.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652608+0000", 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108661 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.702325+0000", 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049717048999999999 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.711665+0000", 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093403370000000006 2024-04-20T02:53:02.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716188+0000", 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004522997 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716227+0000", 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9240999999999999e-05 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716273+0000", 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5228000000000001e-05 2024-04-20T02:53:02.973 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.974 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.974 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.974 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:02.974 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:02.974 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:02.975 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:02.978 INFO:tasks.rados.rados.0.smithi105.stdout:1840: finishing write tid 2 to smithi10548823-33 2024-04-20T02:53:02.978 INFO:tasks.rados.rados.0.smithi105.stdout:1840: finishing write tid 3 to smithi10548823-33 2024-04-20T02:53:02.978 INFO:tasks.rados.rados.0.smithi105.stdout:1840: finishing write tid 4 to smithi10548823-33 2024-04-20T02:53:02.978 INFO:tasks.rados.rados.0.smithi105.stdout:1840: oid 33 updating version 0 to 108 2024-04-20T02:53:02.978 INFO:tasks.rados.rados.0.smithi105.stdout:1840: oid 33 updating version 108 to 109 2024-04-20T02:53:02.978 INFO:tasks.rados.rados.0.smithi105.stdout:1840: oid 33 updating version 109 to 110 2024-04-20T02:53:02.978 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 110 (ObjNum 374 snap 39 seq_num 374) dirty exists 2024-04-20T02:53:02.978 INFO:tasks.rados.rados.0.smithi105.stdout:1840: left oid 33 (ObjNum 374 snap 39 seq_num 374) 2024-04-20T02:53:02.978 INFO:tasks.rados.rados.0.smithi105.stdout:1842: finishing write tid 1 to smithi10548823-42 2024-04-20T02:53:02.978 INFO:tasks.rados.rados.0.smithi105.stdout:1856: seq_num 377 ranges {736239=470170,1728182=571782} 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1856: writing smithi10548823-34 from 736239 to 1206409 tid 1 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1856: writing smithi10548823-34 from 1728182 to 2299964 tid 2 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1840: done (10 left) 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1844: done (9 left) 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1845: done (8 left) 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1846: done (7 left) 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1847: done (6 left) 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1848: done (5 left) 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1857: read oid 20 snap -1 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1857: expect (ObjNum 326 snap 35 seq_num 326) 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1858: write oid 47 current snap is 40 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1842: finishing write tid 2 to smithi10548823-42 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1842: finishing write tid 3 to smithi10548823-42 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1842: finishing write tid 4 to smithi10548823-42 2024-04-20T02:53:02.979 INFO:tasks.rados.rados.0.smithi105.stdout:1842: oid 42 updating version 0 to 95 2024-04-20T02:53:02.980 INFO:tasks.rados.rados.0.smithi105.stdout:1842: oid 42 updating version 95 to 96 2024-04-20T02:53:02.980 INFO:tasks.rados.rados.0.smithi105.stdout:1842: oid 42 version 96 is already newer than 94 2024-04-20T02:53:02.980 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 96 (ObjNum 375 snap 39 seq_num 375) dirty exists 2024-04-20T02:53:02.980 INFO:tasks.rados.rados.0.smithi105.stdout:1842: left oid 42 (ObjNum 375 snap 39 seq_num 375) 2024-04-20T02:53:02.980 INFO:tasks.rados.rados.0.smithi105.stdout:1858: seq_num 378 ranges {753046=545419,1786664=431793} 2024-04-20T02:53:02.980 INFO:tasks.rados.rados.0.smithi105.stdout:1858: writing smithi10548823-47 from 753046 to 1298465 tid 1 2024-04-20T02:53:02.980 INFO:tasks.rados.rados.0.smithi105.stdout:1858: writing smithi10548823-47 from 1786664 to 2218457 tid 2 2024-04-20T02:53:02.980 INFO:tasks.rados.rados.0.smithi105.stdout:1842: done (6 left) 2024-04-20T02:53:02.980 INFO:tasks.rados.rados.0.smithi105.stdout:1859: tier_promote oid 21 2024-04-20T02:53:02.980 INFO:tasks.rados.rados.0.smithi105.stdout:1853: expect (ObjNum 342 snap 37 seq_num 342) 2024-04-20T02:53:02.981 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:02.976+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:03.039 INFO:tasks.rados.rados.0.smithi105.stdout:1852: finishing write tid 1 to smithi10548823-48 2024-04-20T02:53:03.039 INFO:tasks.rados.rados.0.smithi105.stdout:1852: finishing write tid 2 to smithi10548823-48 2024-04-20T02:53:03.039 INFO:tasks.rados.rados.0.smithi105.stdout:1853: done (6 left) 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:1854: done (5 left) 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:1860: snap_create 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:1852: finishing write tid 3 to smithi10548823-48 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:1852: finishing write tid 4 to smithi10548823-48 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:1852: oid 48 updating version 0 to 133 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:1852: oid 48 version 133 is already newer than 132 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:1852: oid 48 version 133 is already newer than 131 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 133 (ObjNum 376 snap 40 seq_num 376) dirty exists 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:1852: left oid 48 (ObjNum 376 snap 40 seq_num 376) 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:1859: got (0) Success 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 172 (ObjNum 365 snap 37 seq_num 365) dirty exists 2024-04-20T02:53:03.040 INFO:tasks.rados.rados.0.smithi105.stdout:1857: expect (ObjNum 326 snap 35 seq_num 326) 2024-04-20T02:53:03.076 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:03.076+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:03.115 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:03.115 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:03.115 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:03.115 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:03.125 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:03.165 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:53:03.210 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:03.210+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:03.219 INFO:tasks.rados.rados.0.smithi105.stdout:1856: finishing write tid 1 to smithi10548823-34 2024-04-20T02:53:03.219 INFO:tasks.rados.rados.0.smithi105.stdout:1856: finishing write tid 2 to smithi10548823-34 2024-04-20T02:53:03.219 INFO:tasks.rados.rados.0.smithi105.stdout:1856: finishing write tid 3 to smithi10548823-34 2024-04-20T02:53:03.220 INFO:tasks.rados.rados.0.smithi105.stdout:1856: finishing write tid 4 to smithi10548823-34 2024-04-20T02:53:03.220 INFO:tasks.rados.rados.0.smithi105.stdout:1856: oid 34 updating version 0 to 277 2024-04-20T02:53:03.220 INFO:tasks.rados.rados.0.smithi105.stdout:1856: oid 34 version 277 is already newer than 275 2024-04-20T02:53:03.220 INFO:tasks.rados.rados.0.smithi105.stdout:1856: oid 34 version 277 is already newer than 276 2024-04-20T02:53:03.220 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 277 (ObjNum 377 snap 40 seq_num 377) dirty exists 2024-04-20T02:53:03.220 INFO:tasks.rados.rados.0.smithi105.stdout:1856: left oid 34 (ObjNum 377 snap 40 seq_num 377) 2024-04-20T02:53:03.221 INFO:tasks.rados.rados.0.smithi105.stdout:1858: finishing write tid 1 to smithi10548823-47 2024-04-20T02:53:03.221 INFO:tasks.rados.rados.0.smithi105.stdout:1858: finishing write tid 2 to smithi10548823-47 2024-04-20T02:53:03.221 INFO:tasks.rados.rados.0.smithi105.stdout:1858: finishing write tid 3 to smithi10548823-47 2024-04-20T02:53:03.221 INFO:tasks.rados.rados.0.smithi105.stdout:1858: finishing write tid 4 to smithi10548823-47 2024-04-20T02:53:03.221 INFO:tasks.rados.rados.0.smithi105.stdout:1858: oid 47 updating version 0 to 164 2024-04-20T02:53:03.221 INFO:tasks.rados.rados.0.smithi105.stdout:1858: oid 47 version 164 is already newer than 163 2024-04-20T02:53:03.221 INFO:tasks.rados.rados.0.smithi105.stdout:1858: oid 47 version 164 is already newer than 162 2024-04-20T02:53:03.222 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 47 v 164 (ObjNum 378 snap 40 seq_num 378) dirty exists 2024-04-20T02:53:03.222 INFO:tasks.rados.rados.0.smithi105.stdout:1858: left oid 47 (ObjNum 378 snap 40 seq_num 378) 2024-04-20T02:53:03.222 INFO:tasks.rados.rados.0.smithi105.stdout:1852: done (5 left) 2024-04-20T02:53:03.222 INFO:tasks.rados.rados.0.smithi105.stdout:1856: done (4 left) 2024-04-20T02:53:03.222 INFO:tasks.rados.rados.0.smithi105.stdout:1857: done (3 left) 2024-04-20T02:53:03.222 INFO:tasks.rados.rados.0.smithi105.stdout:1858: done (2 left) 2024-04-20T02:53:03.223 INFO:tasks.rados.rados.0.smithi105.stdout:1859: done (1 left) 2024-04-20T02:53:03.223 INFO:tasks.rados.rados.0.smithi105.stdout:1860: done (0 left) 2024-04-20T02:53:03.223 INFO:tasks.rados.rados.0.smithi105.stdout:1861: read oid 41 snap -1 2024-04-20T02:53:03.223 INFO:tasks.rados.rados.0.smithi105.stdout:1861: expect (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:53:03.223 INFO:tasks.rados.rados.0.smithi105.stdout:1862: delete oid 27 current snap is 41 2024-04-20T02:53:03.224 INFO:tasks.rados.rados.0.smithi105.stdout:1861: expect (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:53:03.256 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:03.256 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:03.256 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:03.256 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:03.256 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:03.265 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:03.276 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:03.277+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:03.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1861: done (1 left) 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1862: done (0 left) 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1863: read oid 23 snap -1 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1863: expect (ObjNum 373 snap 38 seq_num 373) 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1864: read oid 28 snap -1 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1864: expect (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1865: read oid 7 snap -1 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1865: expect (ObjNum 359 snap 37 seq_num 359) 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1866: read oid 43 snap -1 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1866: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:53:03.342 INFO:tasks.rados.rados.0.smithi105.stdout:1867: write oid 33 current snap is 41 2024-04-20T02:53:03.343 INFO:tasks.rados.rados.0.smithi105.stdout:1867: seq_num 380 ranges {786660=695917,1903628=435462} 2024-04-20T02:53:03.351 INFO:tasks.rados.rados.0.smithi105.stdout:1867: writing smithi10548823-33 from 786660 to 1482577 tid 1 2024-04-20T02:53:03.356 INFO:tasks.rados.rados.0.smithi105.stdout:1867: writing smithi10548823-33 from 1903628 to 2339090 tid 2 2024-04-20T02:53:03.356 INFO:tasks.rados.rados.0.smithi105.stdout:1868: read oid 14 snap -1 2024-04-20T02:53:03.356 INFO:tasks.rados.rados.0.smithi105.stdout:1868: expect (ObjNum 343 snap 37 seq_num 343) 2024-04-20T02:53:03.356 INFO:tasks.rados.rados.0.smithi105.stdout:1869: snap_remove snap 40 2024-04-20T02:53:03.377 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:03.377+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:03.405 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.66917431799999, 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:03.406 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:03.407 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:03.407 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:53:03.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.407 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:03.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:03.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:53:03.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:03.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:53:03.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:53:03.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:53:03.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:03.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:53:03.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:53:03.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:53:03.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.63788525699999, 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:03.413 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:53:03.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:03.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:53:03.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:53:03.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:53:03.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:53:03.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:03.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:53:03.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:53:03.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:03.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:53:03.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:53:03.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:53:03.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:53:03.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:53:03.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:53:03.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.633194801, 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:03.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:03.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:53:03.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:53:03.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:53:03.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:03.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:53:03.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:53:03.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:53:03.426 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:53:03.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:53:03.427 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:53:03.427 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.427 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.427 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.427 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.62643833199999, 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:53:03.428 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.429 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.430 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.431 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:53:03.432 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:53:03.433 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.61662282099999, 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:53:03.434 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.435 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.436 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.437 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:53:03.438 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:53:03.439 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.61557888900001, 2024-04-20T02:53:03.440 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.441 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.442 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.443 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.444 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.445 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.61555772899999, 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.446 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.447 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.448 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:03.449 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.450 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.40798714799999, 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.451 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.452 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.453 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.454 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:03.455 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.39379250299999, 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.456 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:53:03.457 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:53:03.458 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.459 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.38782524600001, 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.460 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.461 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.462 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:53:03.463 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.379419243, 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.464 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.465 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:53:03.466 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:53:03.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.275787202, 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:53:03.468 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:53:03.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.472 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.04519089199999, 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.476 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 141.01824674299999, 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:53:03.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.480 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.99288847599999, 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.96474803199999, 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.485 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:03.489 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.92737942700001, 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:53:03.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:53:03.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.927082504, 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:53:03.497 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.92199938600001, 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.894996827, 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:03.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:03.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:53:03.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:53:03.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:03.507 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:03.508 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:03.509 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:03.509 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:03.509 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:03.509 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:03.509+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:03.510 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:03.520 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:03.540 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:03.559 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:03.578 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:03.578 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:03.578 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:03.578 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:03.578 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:03.578 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-20T02:53:03.578 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:03.578 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:03.578 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:53:03.581 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:53:03.622 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:03.622+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:03.641 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:03.642 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:03.652 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:03.711 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:03.711+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:03.789 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:03.789 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:03.802 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:03.846 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:03.847+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:03.914 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:03.914+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:03.932 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:03.932 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:03.942 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:04.015 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:04.015+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:04.111 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:04.111 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:04.117 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:04.117+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:04.122 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:04.171 INFO:tasks.rados.rados.0.smithi105.stdout:1869: done (6 left) 2024-04-20T02:53:04.171 INFO:tasks.rados.rados.0.smithi105.stdout:1870: read oid 5 snap 39 2024-04-20T02:53:04.171 INFO:tasks.rados.rados.0.smithi105.stdout:1870: expect (ObjNum 368 snap 37 seq_num 368) 2024-04-20T02:53:04.171 INFO:tasks.rados.rados.0.smithi105.stdout:1871: read oid 18 snap -1 2024-04-20T02:53:04.172 INFO:tasks.rados.rados.0.smithi105.stdout:1871: expect (ObjNum 370 snap 37 seq_num 370) 2024-04-20T02:53:04.172 INFO:tasks.rados.rados.0.smithi105.stdout:1872: set_chunk oid 29 target oid 2024-04-20T02:53:04.177 INFO:tasks.rados.rados.0.smithi105.stdout:1872: set_chunk oid 29 offset: 168960 length: 113664 target oid 6b62de425d273e324c7211fc007f3f5eab10f631b20859541c4d41a7e026019a offset: 0 snap -1 2024-04-20T02:53:04.177 INFO:tasks.rados.rados.0.smithi105.stdout:1873: write (excl) oid 26 current snap is 41 2024-04-20T02:53:04.177 INFO:tasks.rados.rados.0.smithi105.stdout:1873: seq_num 381 ranges {403467=487519,1562110=579120} 2024-04-20T02:53:04.181 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:53:04.182 INFO:tasks.rados.rados.0.smithi105.stdout:1873: writing smithi10548823-26 from 403467 to 890986 tid 1 2024-04-20T02:53:04.189 INFO:tasks.rados.rados.0.smithi105.stdout:1873: writing smithi10548823-26 from 1562110 to 2141230 tid 2 2024-04-20T02:53:04.189 INFO:tasks.rados.rados.0.smithi105.stdout:1874: read oid 41 snap 38 2024-04-20T02:53:04.190 INFO:tasks.rados.rados.0.smithi105.stdout:1874: expect (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:53:04.190 INFO:tasks.rados.rados.0.smithi105.stdout:1875: read oid 11 snap -1 2024-04-20T02:53:04.190 INFO:tasks.rados.rados.0.smithi105.stdout:1875: expect (ObjNum 347 snap 37 seq_num 347) 2024-04-20T02:53:04.190 INFO:tasks.rados.rados.0.smithi105.stdout:1876: write (excl) oid 21 current snap is 41 2024-04-20T02:53:04.190 INFO:tasks.rados.rados.0.smithi105.stdout:1876: seq_num 382 ranges {420274=562768,1620592=439131} 2024-04-20T02:53:04.195 INFO:tasks.rados.rados.0.smithi105.stdout:1876: writing smithi10548823-21 from 420274 to 983042 tid 1 2024-04-20T02:53:04.202 INFO:tasks.rados.rados.0.smithi105.stdout:1876: writing smithi10548823-21 from 1620592 to 2059723 tid 2 2024-04-20T02:53:04.202 INFO:tasks.rados.rados.0.smithi105.stdout:1877: write (excl) oid 25 current snap is 41 2024-04-20T02:53:04.202 INFO:tasks.rados.rados.0.smithi105.stdout:1877: seq_num 383 ranges {437081=638017,1679074=582789} 2024-04-20T02:53:04.209 INFO:tasks.rados.rados.0.smithi105.stdout:1877: writing smithi10548823-25 from 437081 to 1075098 tid 1 2024-04-20T02:53:04.217 INFO:tasks.rados.rados.0.smithi105.stdout:1877: writing smithi10548823-25 from 1679074 to 2261863 tid 2 2024-04-20T02:53:04.217 INFO:tasks.rados.rados.0.smithi105.stdout:1878: rollback oid 44 current snap is 41 2024-04-20T02:53:04.217 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 44 to 38 2024-04-20T02:53:04.217 INFO:tasks.rados.rados.0.smithi105.stdout:1879: tier_promote oid 37 2024-04-20T02:53:04.217 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:53:04.218 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:04.218+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:04.218 INFO:tasks.rados.rados.0.smithi105.stdout:1865: expect (ObjNum 359 snap 37 seq_num 359) 2024-04-20T02:53:04.267 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:04.267 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:04.278 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:04.318 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:04.319+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:04.325 INFO:tasks.rados.rados.0.smithi105.stdout:1863: expect (ObjNum 373 snap 38 seq_num 373) 2024-04-20T02:53:04.406 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:04.407 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:04.417 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:04.420 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:04.419+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:04.477 INFO:tasks.rados.rados.0.smithi105.stdout:1866: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:53:04.515 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:04.516+0000 7f9d41cb77c0 -1 osd.4 149 log_to_monitors true 2024-04-20T02:53:04.519 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:04.520+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:04.547 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:04.547 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:04.558 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:04.621 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:04.621+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:04.685 INFO:tasks.rados.rados.0.smithi105.stdout:1864: expect (ObjNum 308 snap 33 seq_num 308) 2024-04-20T02:53:04.687 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:04.687 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:04.698 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:04.722 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:04.722+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:04.822 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:04.822+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:04.868 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:04.868 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:04.879 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:04.911 INFO:tasks.rados.rados.0.smithi105.stdout:1868: expect (ObjNum 343 snap 37 seq_num 343) 2024-04-20T02:53:04.924 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:04.924+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:05.024 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:05.024+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:05.039 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:05.039 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:05.050 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:05.118 INFO:tasks.rados.rados.0.smithi105.stdout:1867: finishing write tid 1 to smithi10548823-33 2024-04-20T02:53:05.118 INFO:tasks.rados.rados.0.smithi105.stdout:1867: finishing write tid 2 to smithi10548823-33 2024-04-20T02:53:05.118 INFO:tasks.rados.rados.0.smithi105.stdout:1867: finishing write tid 3 to smithi10548823-33 2024-04-20T02:53:05.118 INFO:tasks.rados.rados.0.smithi105.stdout:1867: finishing write tid 4 to smithi10548823-33 2024-04-20T02:53:05.118 INFO:tasks.rados.rados.0.smithi105.stdout:1867: oid 33 updating version 0 to 112 2024-04-20T02:53:05.118 INFO:tasks.rados.rados.0.smithi105.stdout:1867: oid 33 updating version 112 to 114 2024-04-20T02:53:05.118 INFO:tasks.rados.rados.0.smithi105.stdout:1867: oid 33 version 114 is already newer than 113 2024-04-20T02:53:05.118 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 33 v 114 (ObjNum 380 snap 41 seq_num 380) dirty exists 2024-04-20T02:53:05.118 INFO:tasks.rados.rados.0.smithi105.stdout:1867: left oid 33 (ObjNum 380 snap 41 seq_num 380) 2024-04-20T02:53:05.119 INFO:tasks.rados.rados.0.smithi105.stdout:1871: expect (ObjNum 370 snap 37 seq_num 370) 2024-04-20T02:53:05.124 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:05.125+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:05.186 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:05.187+0000 7f9d31c51700 -1 osd.4 149 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-20T02:53:05.221 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:05.222 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:05.226 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:05.227+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:05.232 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:05.242 INFO:tasks.rados.rados.0.smithi105.stdout:1872: finishing set_chunk to oid 29 2024-04-20T02:53:05.242 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk returned error code -95 offset: 168960 length: 113664 2024-04-20T02:53:05.242 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 364 snap 37 seq_num 364) dirty exists 2024-04-20T02:53:05.242 INFO:tasks.rados.rados.0.smithi105.stdout:1870: expect (ObjNum 368 snap 37 seq_num 368) 2024-04-20T02:53:05.327 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:05.327+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:05.371 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:05.371 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:05.381 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:05.420 INFO:tasks.rados.rados.0.smithi105.stdout:1875: expect (ObjNum 347 snap 37 seq_num 347) 2024-04-20T02:53:05.427 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:05.428+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:05.528 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:05.528+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:05.547 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:05.547 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:05.558 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:05.575 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:05.582 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:05.642 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:05.642+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:05.654 INFO:tasks.rados.rados.0.smithi105.stdout:1874: expect (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:53:05.717 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:05.717 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:05.717 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:05.718 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:05.728 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-04-20T02:53:05.730 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:05.730+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:05.734 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:05.734 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:05.745 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:05.762 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:05.784 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:05.786 INFO:tasks.rados.rados.0.smithi105.stdout:1873: finishing write tid 1 to smithi10548823-26 2024-04-20T02:53:05.786 INFO:tasks.rados.rados.0.smithi105.stdout:1873: finishing write tid 2 to smithi10548823-26 2024-04-20T02:53:05.786 INFO:tasks.rados.rados.0.smithi105.stdout:1863: done (15 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1864: done (14 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1865: done (13 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1866: done (12 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1867: done (11 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1868: done (10 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1870: done (9 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1871: done (8 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1872: done (7 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1874: done (6 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1875: done (5 left) 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1880: read oid 38 snap 39 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1880: expect deleted 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1881: delete oid 24 current snap is 41 2024-04-20T02:53:05.787 INFO:tasks.rados.rados.0.smithi105.stdout:1879: got (0) Success 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 92 (ObjNum 327 snap 35 seq_num 327) dirty exists 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1873: finishing write tid 3 to smithi10548823-26 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1873: finishing write tid 4 to smithi10548823-26 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1873: oid 26 updating version 0 to 446 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1873: oid 26 version 446 is already newer than 444 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1873: oid 26 version 446 is already newer than 445 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 446 (ObjNum 381 snap 41 seq_num 381) dirty exists 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1873: left oid 26 (ObjNum 381 snap 41 seq_num 381) 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1878: finishing rollback tid 0 to smithi10548823-44 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1876: finishing write tid 1 to smithi10548823-21 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1876: finishing write tid 2 to smithi10548823-21 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1878: finishing rollback tid 1 to smithi10548823-44 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1876: finishing write tid 3 to smithi10548823-21 2024-04-20T02:53:05.788 INFO:tasks.rados.rados.0.smithi105.stdout:1878: finishing rollback tid 2 to smithi10548823-44 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 145 (ObjNum 208 snap 19 seq_num 208) dirty exists 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1876: finishing write tid 4 to smithi10548823-21 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1876: oid 21 updating version 0 to 176 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1876: oid 21 version 176 is already newer than 175 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1876: oid 21 updating version 176 to 177 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 177 (ObjNum 382 snap 41 seq_num 382) dirty exists 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1876: left oid 21 (ObjNum 382 snap 41 seq_num 382) 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1877: finishing write tid 1 to smithi10548823-25 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1877: finishing write tid 2 to smithi10548823-25 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1877: finishing write tid 3 to smithi10548823-25 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1877: finishing write tid 4 to smithi10548823-25 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1877: oid 25 updating version 0 to 291 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1877: oid 25 updating version 291 to 292 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:1877: oid 25 updating version 292 to 293 2024-04-20T02:53:05.789 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 293 (ObjNum 383 snap 41 seq_num 383) dirty exists 2024-04-20T02:53:05.790 INFO:tasks.rados.rados.0.smithi105.stdout:1877: left oid 25 (ObjNum 383 snap 41 seq_num 383) 2024-04-20T02:53:05.790 INFO:tasks.rados.rados.0.smithi105.stdout:1873: done (6 left) 2024-04-20T02:53:05.790 INFO:tasks.rados.rados.0.smithi105.stdout:1876: done (5 left) 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1877: done (4 left) 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1878: done (3 left) 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1879: done (2 left) 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1880: done (1 left) 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1881: done (0 left) 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1882: read oid 26 snap -1 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1882: expect (ObjNum 381 snap 41 seq_num 381) 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1883: read oid 21 snap 39 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1883: expect (ObjNum 365 snap 37 seq_num 365) 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1884: write (excl) oid 36 current snap is 41 2024-04-20T02:53:05.791 INFO:tasks.rados.rados.0.smithi105.stdout:1884: seq_num 385 ranges {470695=788515,1679685=586458} 2024-04-20T02:53:05.796 INFO:tasks.rados.rados.0.smithi105.stdout:1884: writing smithi10548823-36 from 470695 to 1259210 tid 1 2024-04-20T02:53:05.803 INFO:tasks.rados.rados.0.smithi105.stdout:1884: writing smithi10548823-36 from 1679685 to 2266143 tid 2 2024-04-20T02:53:05.803 INFO:tasks.rados.rados.0.smithi105.stdout:1885: set_chunk oid 12 target oid 2024-04-20T02:53:05.803 INFO:tasks.rados.rados.0.smithi105.stdout:1885: set_chunk oid 12 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:53:05.803 INFO:tasks.rados.rados.0.smithi105.stdout:1886: read oid 16 snap 39 2024-04-20T02:53:05.804 INFO:tasks.rados.rados.0.smithi105.stdout:1886: expect deleted 2024-04-20T02:53:05.804 INFO:tasks.rados.rados.0.smithi105.stdout:1887: snap_remove snap 38 2024-04-20T02:53:05.812 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:05.831 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:05.832+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:05.833 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:05.858 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:05.877 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:05.897 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:05.897 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:05.897 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:05.897 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:05.897 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:05.897 INFO:teuthology.orchestra.run.smithi105.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-04-20T02:53:05.897 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:05.897 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:05.897 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-20T02:53:05.900 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-20T02:53:05.911 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:05.911 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:05.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:05.911 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:05.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.911 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:05.911 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:05.911 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 139.325094882, 2024-04-20T02:53:05.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:53:05.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:53:05.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:05.917 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 139.30436557100001, 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:05.923 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 139.29511382000001, 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:05.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:53:05.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:53:05.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:05.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 131.49811460399999, 2024-04-20T02:53:05.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:53:05.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:53:05.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 131.49210981600001, 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.936 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.937 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4029 4.13 4:cf11ccc0:::smithi10548823-4:head [write 699257~564637,stat] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.354567299999999, 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.86467550400000004, 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4029 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.938 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555666+0000", 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:05.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555809+0000", 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014306900000000001 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555832+0000", 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3680999999999999e-05 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555846+0000", 2024-04-20T02:53:05.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3654e-05 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555913+0000", 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6736999999999995e-05 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556058+0000", 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014538100000000001 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:05.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556392+0000", 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033393099999999998 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578661+0000", 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022268553999999999 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578702+0000", 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1533999999999999e-05 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578747+0000", 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4858999999999999e-05 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578811+0000", 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3520000000000002e-05 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605743+0000", 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026932418999999999 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420312+0000", 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81456849600000003 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420343+0000", 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1911e-05 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4030 4.13 4:cf11ccc0:::smithi10548823-4:head [write 1873062~640438] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:05.944 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.347654178999999, 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85780235299999996, 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4030 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562579+0000", 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562714+0000", 2024-04-20T02:53:05.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013475 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562728+0000", 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3516000000000001e-05 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562736+0000", 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5690000000000005e-06 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578835+0000", 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016099156 2024-04-20T02:53:05.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578957+0000", 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121094 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.579123+0000", 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016647100000000001 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605727+0000", 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026603643999999999 2024-04-20T02:53:05.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605749+0000", 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2070999999999999e-05 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605799+0000", 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0417000000000002e-05 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605841+0000", 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2344e-05 2024-04-20T02:53:05.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605907+0000", 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5135999999999994e-05 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420373+0000", 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81446675899999998 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420383+0000", 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0108999999999999e-05 2024-04-20T02:53:05.950 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4031 4.13 4:cf11ccc0:::smithi10548823-4:head [write 3305941~593316] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.340092603999999, 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85027709799999995, 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.951 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4031 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570142+0000", 2024-04-20T02:53:05.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570277+0000", 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135606 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570288+0000", 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0719e-05 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570294+0000", 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2519999999999996e-06 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592786+0000", 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022491220999999999 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592911+0000", 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012546599999999999 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:05.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.593118+0000", 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000207011 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605878+0000", 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012760298999999999 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605934+0000", 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5813999999999998e-05 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605963+0000", 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9275e-05 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606612+0000", 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00064813999999999996 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613782+0000", 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071705670000000001 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420411+0000", 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.80662907399999995 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420420+0000", 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5790000000000004e-06 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4032 4.13 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (56) in=7b,truncate 3899257] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.339936175999998, 2024-04-20T02:53:05.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85015831100000006, 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4032 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570302+0000", 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1700000000000001e-06 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570305+0000", 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.976e-06 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570308+0000", 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8899999999999999e-06 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605985+0000", 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035677121999999999 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606066+0000", 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0797000000000006e-05 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606211+0000", 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014569999999999999 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607013+0000", 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00080160599999999995 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607046+0000", 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2938999999999998e-05 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607086+0000", 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0203000000000001e-05 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607157+0000", 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0621e-05 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613833+0000", 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066762269999999999 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420448+0000", 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.806615478 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420457+0000", 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9779999999999994e-06 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.963 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4033 4.13 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.339924302, 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85021192700000003, 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4033 2024-04-20T02:53:05.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570312+0000", 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419999999999999e-06 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570316+0000", 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3730000000000001e-06 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570318+0000", 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6680000000000001e-06 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606541+0000", 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036222797000000001 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606598+0000", 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6629999999999998e-05 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420515+0000", 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81391740099999998 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420523+0000", 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6799999999999993e-06 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4034 4.13 4:cfd1e4d4:::smithi10548823-9:head [rollback 3f] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:05.968 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.339914663999998, 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85017147900000001, 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4034 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570320+0000", 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570322+0000", 2024-04-20T02:53:05.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.37e-06 2024-04-20T02:53:05.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570325+0000", 2024-04-20T02:53:05.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7800000000000001e-06 2024-04-20T02:53:05.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570328+0000", 2024-04-20T02:53:05.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5379999999999998e-06 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606632+0000", 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036304075999999998 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606716+0000", 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3689000000000005e-05 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420484+0000", 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81376812700000001 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420492+0000", 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1179999999999992e-06 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4062 4.9 4:9c87d36d:::smithi10548823-24:head [write 766485~465633] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:05.973 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.760060883, 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075029242999999995, 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4062 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150173+0000", 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150435+0000", 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000262064 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150445+0000", 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8160000000000005e-06 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150450+0000", 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1880000000000001e-06 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150484+0000", 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150490+0000", 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6810000000000003e-06 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161686+0000", 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011196187 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161840+0000", 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015389000000000001 2024-04-20T02:53:05.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.162103+0000", 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026333299999999998 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224927+0000", 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062823458999999998 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224993+0000", 2024-04-20T02:53:05.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6667000000000004e-05 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225059+0000", 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6117000000000004e-05 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225119+0000", 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9342000000000002e-05 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225204+0000", 2024-04-20T02:53:05.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4844000000000003e-05 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4063 4.9 4:9c87d36d:::smithi10548823-24:head [write 1990637~647776] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.759778209, 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075097435000000004, 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.980 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4063 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150746+0000", 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028951399999999999 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150753+0000", 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150757+0000", 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2300000000000002e-06 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150801+0000", 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3421000000000003e-05 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150811+0000", 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0581000000000001e-05 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177065+0000", 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02625367 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177164+0000", 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8900999999999994e-05 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177319+0000", 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000155437 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225048+0000", 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047728857 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225217+0000", 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168395 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225375+0000", 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157837 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225437+0000", 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2824999999999996e-05 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225555+0000", 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117169 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4064 4.9 4:9c87d36d:::smithi10548823-24:head [write 3299280~667205] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:05.986 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.759471456, 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112222372, 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4064 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150763+0000", 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151076+0000", 2024-04-20T02:53:05.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000312355 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151083+0000", 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1010000000000003e-06 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151091+0000", 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0409999999999998e-06 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151127+0000", 2024-04-20T02:53:05.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151133+0000", 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1099999999999999e-06 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200197+0000", 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049064000000000003 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200315+0000", 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117855 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200545+0000", 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022990700000000001 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225595+0000", 2024-04-20T02:53:05.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025050224999999999 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262670+0000", 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075460999999997 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262767+0000", 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6479999999999998e-05 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262835+0000", 2024-04-20T02:53:05.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999999e-05 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262986+0000", 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015129799999999999 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4065 4.9 4:9c87d36d:::smithi10548823-24:head [setxattr _header (57) in=7b,truncate 3966485] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.759131576, 2024-04-20T02:53:05.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112193353, 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4065 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151103+0000", 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151107+0000", 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:05.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151111+0000", 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2309999999999998e-06 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151116+0000", 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4850000000000003e-06 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151168+0000", 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2673000000000002e-05 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151179+0000", 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0112000000000001e-05 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225611+0000", 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074432549000000001 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:05.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225689+0000", 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7612000000000002e-05 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225858+0000", 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016945400000000001 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262650+0000", 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036792126000000001 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262678+0000", 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8132000000000001e-05 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263077+0000", 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039868299999999999 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263141+0000", 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3448999999999995e-05 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:05.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263297+0000", 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015649699999999999 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4066 4.9 4:9c87d36d:::smithi10548823-24:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.759114719999999, 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112663448, 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:06.000 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4066 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151120+0000", 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151123+0000", 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4389999999999999e-06 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151128+0000", 2024-04-20T02:53:06.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2449999999999998e-06 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151132+0000", 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9049999999999999e-06 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151198+0000", 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5988000000000006e-05 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151205+0000", 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8770000000000004e-06 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226171+0000", 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074966844000000005 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226261+0000", 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9975999999999994e-05 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263337+0000", 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075863000000001 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263443+0000", 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000105678 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263784+0000", 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034103200000000001 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.005 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.006 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.006 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4095 4.9 4:964390c0:::smithi10548823-36:head [write 450520~558231] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.006 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:06.006 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.759100650000001, 2024-04-20T02:53:06.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112875926, 2024-04-20T02:53:06.006 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.006 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4095 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151134+0000", 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151379+0000", 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024486000000000001 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151385+0000", 2024-04-20T02:53:06.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1460000000000001e-06 2024-04-20T02:53:06.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151389+0000", 2024-04-20T02:53:06.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6610000000000002e-06 2024-04-20T02:53:06.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151413+0000", 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3611000000000001e-05 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151419+0000", 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8780000000000003e-06 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226302+0000", 2024-04-20T02:53:06.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074883540999999998 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226460+0000", 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015792000000000001 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226599+0000", 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013939599999999999 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262685+0000", 2024-04-20T02:53:06.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036086115000000002 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262726+0000", 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0448999999999998e-05 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263833+0000", 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001106685 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263884+0000", 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1530999999999998e-05 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264011+0000", 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012641900000000001 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.013 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4096 4.9 4:964390c0:::smithi10548823-36:head [write 1483047~515125] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.758841114999999, 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11703358799999999, 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4096 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151644+0000", 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025029799999999998 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151651+0000", 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.148e-06 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151655+0000", 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2769999999999999e-06 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151689+0000", 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711999999999998e-05 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151698+0000", 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0359999999999995e-06 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245111+0000", 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093413164000000007 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245232+0000", 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000120214 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245424+0000", 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019272500000000001 2024-04-20T02:53:06.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263033+0000", 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017608461999999998 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266386+0000", 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033530980000000001 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268218+0000", 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001832101 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268283+0000", 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5334999999999997e-05 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268428+0000", 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014449999999999999 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4097 4.9 4:964390c0:::smithi10548823-36:head [setxattr _header (57) in=7b,truncate 2050520] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.758573161999999, 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.117031099, 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4097 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151665+0000", 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.614e-06 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151670+0000", 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151675+0000", 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0560000000000004e-06 2024-04-20T02:53:06.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151738+0000", 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2447999999999996e-05 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151753+0000", 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5007e-05 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264044+0000", 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112291562 2024-04-20T02:53:06.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264165+0000", 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121081 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264421+0000", 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025568599999999998 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266133+0000", 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001711668 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266755+0000", 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00062224100000000005 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268484+0000", 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001729169 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268545+0000", 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0365000000000002e-05 2024-04-20T02:53:06.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268693+0000", 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014860000000000001 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4098 4.9 4:964390c0:::smithi10548823-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 13.758555680000001, 2024-04-20T02:53:06.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11735205, 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4098 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151679+0000", 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151681+0000", 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3140000000000002e-06 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151686+0000", 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0209999999999999e-06 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151691+0000", 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4700000000000004e-06 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151779+0000", 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8399999999999994e-05 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151788+0000", 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7129999999999994e-06 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265294+0000", 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11350631899999999 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265391+0000", 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6861000000000007e-05 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268730+0000", 2024-04-20T02:53:06.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003339102 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268841+0000", 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110708 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.269032+0000", 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019057899999999999 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.032 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.033 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:06.034 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:06.035 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:06.035+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:06.135 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:06.135+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:06.177 INFO:tasks.rados.rados.0.smithi105.stdout:1887: done (5 left) 2024-04-20T02:53:06.178 INFO:tasks.rados.rados.0.smithi105.stdout:1888: write (excl) oid 40 current snap is 41 2024-04-20T02:53:06.178 INFO:tasks.rados.rados.0.smithi105.stdout:1883: expect (ObjNum 365 snap 37 seq_num 365) 2024-04-20T02:53:06.202 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:53:06.217 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:06.217 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:06.217 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:06.217 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:06.229 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:06.270 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:06.270+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:06.294 INFO:tasks.rados.rados.0.smithi105.stdout:1882: expect (ObjNum 381 snap 41 seq_num 381) 2024-04-20T02:53:06.335 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:06.336+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:06.404 INFO:tasks.rados.rados.0.smithi105.stdout:1885: finishing set_chunk to oid 12 2024-04-20T02:53:06.404 INFO:tasks.rados.rados.0.smithi105.stdout:1885: got expected ERANGE (src dne) 2024-04-20T02:53:06.405 INFO:tasks.rados.rados.0.smithi105.stdout:1888: seq_num 386 ranges {487502=463764,1738167=446469} 2024-04-20T02:53:06.409 INFO:tasks.rados.rados.0.smithi105.stdout:1888: writing smithi10548823-40 from 487502 to 951266 tid 1 2024-04-20T02:53:06.409 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:06.409 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:06.409 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:06.409 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:06.410 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:06.417 INFO:tasks.rados.rados.0.smithi105.stdout:1888: writing smithi10548823-40 from 1738167 to 2184636 tid 2 2024-04-20T02:53:06.417 INFO:tasks.rados.rados.0.smithi105.stdout:1882: done (5 left) 2024-04-20T02:53:06.417 INFO:tasks.rados.rados.0.smithi105.stdout:1883: done (4 left) 2024-04-20T02:53:06.417 INFO:tasks.rados.rados.0.smithi105.stdout:1885: done (3 left) 2024-04-20T02:53:06.417 INFO:tasks.rados.rados.0.smithi105.stdout:1886: done (2 left) 2024-04-20T02:53:06.417 INFO:tasks.rados.rados.0.smithi105.stdout:1889: read oid 46 snap -1 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1889: expect (ObjNum 334 snap 36 seq_num 334) 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1890: set_chunk oid 16 target oid 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1890: set_chunk oid 16 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1891: snap_create 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1884: finishing write tid 1 to smithi10548823-36 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1884: finishing write tid 2 to smithi10548823-36 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1884: finishing write tid 3 to smithi10548823-36 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1884: finishing write tid 4 to smithi10548823-36 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1884: oid 36 updating version 0 to 450 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1884: oid 36 updating version 450 to 452 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:1884: oid 36 version 452 is already newer than 451 2024-04-20T02:53:06.418 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 452 (ObjNum 385 snap 41 seq_num 385) dirty exists 2024-04-20T02:53:06.419 INFO:tasks.rados.rados.0.smithi105.stdout:1884: left oid 36 (ObjNum 385 snap 41 seq_num 385) 2024-04-20T02:53:06.419 INFO:tasks.rados.rados.0.smithi105.stdout:1890: finishing set_chunk to oid 16 2024-04-20T02:53:06.419 INFO:tasks.rados.rados.0.smithi105.stdout:1890: got expected ENOENT (src dne) 2024-04-20T02:53:06.424 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:06.426 INFO:tasks.rados.rados.0.smithi105.stdout:1889: expect (ObjNum 334 snap 36 seq_num 334) 2024-04-20T02:53:06.469 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:06.469+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:06.537 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:06.538+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:06.559 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:06.559 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 140.01463694, 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.560 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:53:06.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:53:06.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:53:06.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:53:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 132.186088052, 2024-04-20T02:53:06.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:53:06.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:06.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 132.16983586999999, 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.574 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.575 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:06.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:53:06.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:53:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 74.210136851000001, 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:53:06.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:53:06.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:53:06.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:53:06.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:53:06.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3872 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 779924~785847,stat] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:06.588 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.207733611999998, 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070878165000000007, 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3872 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350641+0000", 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351179+0000", 2024-04-20T02:53:06.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053797199999999995 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351196+0000", 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.66e-05 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351209+0000", 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2952000000000001e-05 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351306+0000", 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7272e-05 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351322+0000", 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6416e-05 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358164+0000", 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068416019999999996 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358336+0000", 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017156499999999999 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358594+0000", 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025794499999999998 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389148+0000", 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030554773 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409688+0000", 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020539497 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420893+0000", 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011205503 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420934+0000", 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1171000000000003e-05 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420998+0000", 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3386999999999995e-05 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421521+0000", 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052313200000000005 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3873 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 2078257~580112] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:06.595 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.207150326000001, 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075737411000000004, 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3873 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351225+0000", 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351614+0000", 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038825099999999999 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351625+0000", 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1173e-05 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351634+0000", 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0489999999999993e-06 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351696+0000", 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1885999999999997e-05 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351708+0000", 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2702e-05 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389193+0000", 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037484574999999999 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389303+0000", 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110188 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389563+0000", 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026034499999999999 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420846+0000", 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031283014999999997 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421592+0000", 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074529299999999995 2024-04-20T02:53:06.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421632+0000", 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0677999999999999e-05 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426683+0000", 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050507119999999997 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426752+0000", 2024-04-20T02:53:06.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8484000000000004e-05 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426964+0000", 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021193500000000001 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3874 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 3301257~278667] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:06.602 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.206715325000001, 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075720365999999997, 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3874 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351660+0000", 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351852+0000", 2024-04-20T02:53:06.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019116900000000001 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351861+0000", 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6050000000000004e-06 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351869+0000", 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4780000000000002e-06 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351916+0000", 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7222999999999998e-05 2024-04-20T02:53:06.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351927+0000", 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0791000000000001e-05 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409721+0000", 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057793887000000002 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409840+0000", 2024-04-20T02:53:06.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119837 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.410063+0000", 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022257200000000001 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421678+0000", 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011615448 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421721+0000", 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2191000000000003e-05 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.423989+0000", 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022682879999999998 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427051+0000", 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003061785 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427157+0000", 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106684 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427381+0000", 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022416299999999999 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3875 4.10 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (57) in=7b,truncate 3579924] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:06.609 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.206494163999999, 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075851731000000006, 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3875 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351886+0000", 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.561e-06 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351893+0000", 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6100000000000002e-06 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351899+0000", 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8660000000000002e-06 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352002+0000", 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103811 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352012+0000", 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4110000000000002e-06 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421749+0000", 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.069736690000000004 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421861+0000", 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112587 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422120+0000", 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025878099999999997 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424547+0000", 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024270379999999999 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424960+0000", 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041259899999999998 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426257+0000", 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001297231 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427489+0000", 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012320949999999999 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427562+0000", 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2825999999999997e-05 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427734+0000", 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017230699999999999 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3876 4.10 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:06.616 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.206472258000002, 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076230629999999994, 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3876 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351906+0000", 2024-04-20T02:53:06.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6350000000000002e-06 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351912+0000", 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8209999999999998e-06 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351918+0000", 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3839999999999996e-06 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352035+0000", 2024-04-20T02:53:06.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117207 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352043+0000", 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0169999999999996e-06 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422511+0000", 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070467815000000003 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422619+0000", 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010819600000000001 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427772+0000", 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005153398 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427900+0000", 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012740399999999999 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.428135+0000", 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235138 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4067 4.c 4:392c1e2f:::smithi10548823-44:head [write 783292~540882] snapc 47=[47,45,3f] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.415780049, 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072910900000000001, 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.622 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4067 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142595+0000", 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142941+0000", 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034585799999999999 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142954+0000", 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3491e-05 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142965+0000", 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1139e-05 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143003+0000", 2024-04-20T02:53:06.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7499000000000001e-05 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143011+0000", 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5490000000000006e-06 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156033+0000", 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013021576999999999 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156242+0000", 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020885500000000001 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156474+0000", 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195900000000001 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177835+0000", 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021361594000000001 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.627 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178072+0000", 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000237304 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.214963+0000", 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036890158999999999 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215046+0000", 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3399000000000006e-05 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215507+0000", 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046127700000000001 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4068 4.c 4:392c1e2f:::smithi10548823-44:head [write 1932766~507787] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.415401106000001, 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073156952999999997, 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.629 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4068 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143300+0000", 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000325225 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143308+0000", 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3860000000000007e-06 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143315+0000", 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7000000000000002e-06 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143360+0000", 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5475000000000001e-05 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143373+0000", 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2204e-05 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178109+0000", 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034736547999999999 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178222+0000", 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112959 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178452+0000", 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000229686 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215864+0000", 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037412038000000002 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215878+0000", 2024-04-20T02:53:06.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4029000000000001e-05 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215929+0000", 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0738000000000001e-05 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215989+0000", 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0350999999999999e-05 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216132+0000", 2024-04-20T02:53:06.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014332099999999999 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4069 4.c 4:392c1e2f:::smithi10548823-44:head [write 3210350~572013] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.415052293, 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108473219, 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.636 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4069 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143664+0000", 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034090500000000002 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143673+0000", 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1119999999999996e-06 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143679+0000", 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3589999999999996e-06 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143724+0000", 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4974e-05 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143734+0000", 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0275000000000001e-05 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197473+0000", 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053738971000000003 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197616+0000", 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014275900000000001 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197834+0000", 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021806000000000001 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251517+0000", 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053683445000000003 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251579+0000", 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1255999999999998e-05 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251590+0000", 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1399e-05 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251657+0000", 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7039999999999995e-05 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251797+0000", 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000140235 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4070 4.c 4:392c1e2f:::smithi10548823-44:head [setxattr _header (57) in=7b,truncate 3983292] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.642 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.414689505, 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108344988, 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4070 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:06.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143686+0000", 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143690+0000", 2024-04-20T02:53:06.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7579999999999998e-06 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143697+0000", 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0119999999999999e-06 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143702+0000", 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.78e-06 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143758+0000", 2024-04-20T02:53:06.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640999999999998e-05 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143766+0000", 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9519999999999994e-06 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216172+0000", 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072406561999999994 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216295+0000", 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122452 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216531+0000", 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023635999999999999 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251815+0000", 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035283760999999997 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251856+0000", 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1270999999999998e-05 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251892+0000", 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251929+0000", 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7055999999999999e-05 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252032+0000", 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103183 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4071 4.c 4:392c1e2f:::smithi10548823-44:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.414654305999999, 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108670215, 2024-04-20T02:53:06.649 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4071 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143721+0000", 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143727+0000", 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2510000000000002e-06 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143739+0000", 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2649e-05 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143749+0000", 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3340000000000008e-06 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143788+0000", 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9773e-05 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143796+0000", 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8720000000000002e-06 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216846+0000", 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073049823 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216948+0000", 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101525 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252070+0000", 2024-04-20T02:53:06.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035122911999999999 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252180+0000", 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010907499999999999 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252392+0000", 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021285400000000001 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4106 4.c 4:3d524298:::smithi10548823-18:head [write 484134~708729] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.414621200999999, 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108955254, 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4106 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144188+0000", 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000433771 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144198+0000", 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4970000000000007e-06 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144205+0000", 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3409999999999997e-06 2024-04-20T02:53:06.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145714+0000", 2024-04-20T02:53:06.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015088110000000001 2024-04-20T02:53:06.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145727+0000", 2024-04-20T02:53:06.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2989e-05 2024-04-20T02:53:06.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217011+0000", 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071283814000000001 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217168+0000", 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015707300000000001 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:06.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217397+0000", 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022881900000000001 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251870+0000", 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034473539999999997 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252459+0000", 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000589151 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252511+0000", 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1282999999999997e-05 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252569+0000", 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8294999999999997e-05 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252711+0000", 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000141435 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.661 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4107 4.c 4:3d524298:::smithi10548823-18:head [write 1883658~200476] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.414159136, 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.10880477099999999, 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4107 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.662 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144216+0000", 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:06.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144346+0000", 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000129732 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144355+0000", 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4300000000000006e-06 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144361+0000", 2024-04-20T02:53:06.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9150000000000001e-06 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145754+0000", 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0013929649999999999 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145763+0000", 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2990000000000002e-06 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243874+0000", 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.098111462999999996 2024-04-20T02:53:06.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243990+0000", 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0001156 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.244226+0000", 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235839 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252760+0000", 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085345439999999998 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252817+0000", 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6842000000000002e-05 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252829+0000", 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1817999999999999e-05 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252884+0000", 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4610000000000001e-05 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253022+0000", 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013853800000000001 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4108 4.c 4:3d524298:::smithi10548823-18:head [setxattr _header (57) in=7b,truncate 2084134] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:06.668 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.414010712, 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.120324168, 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4108 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144365+0000", 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144369+0000", 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8500000000000004e-06 2024-04-20T02:53:06.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144374+0000", 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0080000000000002e-06 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144379+0000", 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8239999999999999e-06 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145790+0000", 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014115449999999999 2024-04-20T02:53:06.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145800+0000", 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0429999999999996e-06 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253101+0000", 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.107301173 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253210+0000", 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109146 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253456+0000", 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024599700000000002 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254623+0000", 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001166718 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.258220+0000", 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003597135 2024-04-20T02:53:06.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264486+0000", 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062665819999999997 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264558+0000", 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1364000000000003e-05 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264690+0000", 2024-04-20T02:53:06.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013227099999999999 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4109 4.c 4:3d524298:::smithi10548823-18:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.413993189999999, 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12051582399999999, 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.675 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4109 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144385+0000", 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2869999999999999e-06 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144392+0000", 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5749999999999997e-06 2024-04-20T02:53:06.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144397+0000", 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3589999999999999e-06 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145829+0000", 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00143191 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:06.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145838+0000", 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254052+0000", 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108214029 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254156+0000", 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010343799999999999 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264726+0000", 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010570144 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264796+0000", 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9886000000000001e-05 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264899+0000", 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103336 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.680 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:4146 3.1 e153/152 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head v 153'228, mlcod=0'0)", 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.376597004000001, 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066047175, 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181778+0000", 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181997+0000", 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021896 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182011+0000", 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3339999999999999e-05 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182021+0000", 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.046e-05 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182068+0000", 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6931999999999997e-05 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182104+0000", 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6321e-05 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247786+0000", 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065681787000000005 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.684 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247827+0000", 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0491000000000002e-05 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4183 3.2 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head [write 0~2392064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 12.906962832, 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064859071000000004, 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:06.685 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4183 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:06.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651396+0000", 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999819 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652285+0000", 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088949500000000002 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652306+0000", 2024-04-20T02:53:06.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0812e-05 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652333+0000", 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7086000000000001e-05 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652373+0000", 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9690000000000001e-05 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:06.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652499+0000", 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012642299999999999 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652608+0000", 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108661 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.702325+0000", 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049717048999999999 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.711665+0000", 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093403370000000006 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716188+0000", 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004522997 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716227+0000", 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9240999999999999e-05 2024-04-20T02:53:06.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:06.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:06.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:06.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716273+0000", 2024-04-20T02:53:06.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5228000000000001e-05 2024-04-20T02:53:06.691 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.691 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.691 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.691 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:06.691 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:06.691 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:06.693 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:06.693 INFO:tasks.rados.rados.0.smithi105.stdout:1888: finishing write tid 1 to smithi10548823-40 2024-04-20T02:53:06.694 INFO:tasks.rados.rados.0.smithi105.stdout:1888: finishing write tid 2 to smithi10548823-40 2024-04-20T02:53:06.694 INFO:tasks.rados.rados.0.smithi105.stdout:1888: finishing write tid 3 to smithi10548823-40 2024-04-20T02:53:06.694 INFO:tasks.rados.rados.0.smithi105.stdout:1888: finishing write tid 4 to smithi10548823-40 2024-04-20T02:53:06.694 INFO:tasks.rados.rados.0.smithi105.stdout:1888: oid 40 updating version 0 to 454 2024-04-20T02:53:06.694 INFO:tasks.rados.rados.0.smithi105.stdout:1888: oid 40 updating version 454 to 455 2024-04-20T02:53:06.694 INFO:tasks.rados.rados.0.smithi105.stdout:1888: oid 40 updating version 455 to 456 2024-04-20T02:53:06.694 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 456 (ObjNum 386 snap 41 seq_num 386) dirty exists 2024-04-20T02:53:06.694 INFO:tasks.rados.rados.0.smithi105.stdout:1888: left oid 40 (ObjNum 386 snap 41 seq_num 386) 2024-04-20T02:53:06.694 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:06.693+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:06.794 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:06.794+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:06.829 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:06.830 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:06.830 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:06.840 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:06.894 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:06.894+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:06.967 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:06.967 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:06.968 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:06.968 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:06.968 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:06.978 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:06.994 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:06.995+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:07.095 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:07.096+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:07.108 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:07.108 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:07.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:07.108 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:07.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.108 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:07.108 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:07.108 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.37233214400001, 2024-04-20T02:53:07.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:53:07.108 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:07.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:53:07.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:53:07.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.34104308299999, 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:53:07.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:53:07.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.336352627, 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.119 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:07.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:53:07.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.124 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.32959615799999, 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:07.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:53:07.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:53:07.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.31978064699999, 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.130 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:53:07.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:53:07.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:53:07.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:53:07.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.318736715, 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:53:07.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.31871555500001, 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:07.146 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.11114497400001, 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:53:07.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:53:07.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.09695032900001, 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.154 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.090983072, 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:53:07.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:53:07.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:53:07.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.082577069, 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.978945028, 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.163 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:53:07.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.74834871799999, 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.167 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.72140456899999, 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:53:07.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:53:07.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.69604630200001, 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:53:07.175 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.66790585800001, 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:07.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:53:07.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.182 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.630537253, 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.184 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:53:07.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:53:07.188 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.63024032999999, 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.192 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.625157212, 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:53:07.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:53:07.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:53:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:53:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:53:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:53:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.59815465299999, 2024-04-20T02:53:07.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:53:07.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:53:07.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.202 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:07.204 INFO:tasks.rados.rados.0.smithi105.stdout:1884: done (4 left) 2024-04-20T02:53:07.204 INFO:tasks.rados.rados.0.smithi105.stdout:1888: done (3 left) 2024-04-20T02:53:07.204 INFO:tasks.rados.rados.0.smithi105.stdout:1889: done (2 left) 2024-04-20T02:53:07.204 INFO:tasks.rados.rados.0.smithi105.stdout:1890: done (1 left) 2024-04-20T02:53:07.204 INFO:tasks.rados.rados.0.smithi105.stdout:1891: done (0 left) 2024-04-20T02:53:07.204 INFO:tasks.rados.rados.0.smithi105.stdout:1892: snap_create 2024-04-20T02:53:07.204 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:53:07.205 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:07.206 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:07.204+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:07.304 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:07.305+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:07.376 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:07.376 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:07.376 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:07.376 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:07.386 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:07.405 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:07.405+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:07.505 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:07.506+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:07.516 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:07.516 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:07.516 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:07.516 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:07.516 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:07.526 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:07.606 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:07.607+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:07.657 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:07.657 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:07.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:07.657 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:07.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.657 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:53:07.657 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:07.657 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 168.052624028, 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:53:07.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:53:07.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:53:07.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:53:07.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 146.17545458800001, 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.664 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:53:07.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:53:07.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:53:07.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:53:07.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.81566893600001, 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:53:07.674 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 145.805305331, 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:53:07.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.678 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 127.515662627, 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:07.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:53:07.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:53:07.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:53:07.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 127.510525517, 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:07.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:53:07.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:53:07.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:53:07.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 127.510252505, 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.696 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 127.51022312000001, 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:07.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:07.701 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 104.822895821, 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:07.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:53:07.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:53:07.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.375415598999993, 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 79.358147165000005, 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:53:07.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 56.185001370000002, 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:07.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:53:07.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:53:07.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3881 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'232, mlcod=141'228)", 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.23787883, 2024-04-20T02:53:07.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050172267999999999, 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418659+0000", 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419496+0000", 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00083715799999999996 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419513+0000", 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7489000000000001e-05 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419524+0000", 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0118e-05 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429881+0000", 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010357762 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429905+0000", 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3546000000000001e-05 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468790+0000", 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038884821 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468832+0000", 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2341e-05 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.722 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3898 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'133, mlcod=133'129)", 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.185719857999999, 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038834664999999997, 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470817+0000", 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471051+0000", 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023384600000000001 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471066+0000", 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4205e-05 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471074+0000", 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4090000000000003e-06 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472816+0000", 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017419429999999999 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472843+0000", 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7296000000000001e-05 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509592+0000", 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036748496999999998 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509653+0000", 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1707999999999999e-05 2024-04-20T02:53:07.726 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3899 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'134, mlcod=133'129)", 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.168453421999999, 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038218993999999999, 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488084+0000", 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488262+0000", 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017768600000000001 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488274+0000", 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2388000000000001e-05 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488283+0000", 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3840000000000003e-06 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489094+0000", 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081045100000000001 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489119+0000", 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4751000000000001e-05 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526264+0000", 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037144912000000002 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526304+0000", 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0513000000000003e-05 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3891 4.7 e146/145 4:e2c0d65b:::smithi10548823-46:head v 146'105, mlcod=143'100)", 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.167473327, 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041132308999999999, 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.731 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489072+0000", 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9539999999999998e-06 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489082+0000", 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0329000000000001e-05 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489089+0000", 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4959999999999999e-06 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489814+0000", 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072423699999999997 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489837+0000", 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3218999999999999e-05 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530157+0000", 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040319762000000002 2024-04-20T02:53:07.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530198+0000", 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0886000000000001e-05 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.7 146/145 [PushOp(4:e2c0d65b:::smithi10548823-46:43, version: 143'96, data_included: [426944~820638,1468778~227938], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:e2c0d65b:::smithi10548823-46:43@143'96, size: 3311012, copy_subset: [0~3311012], clone_subset: {}, snapset: 45=[]:{3f=[3f,39],43=[41],45=[45]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1696716, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.166261975000001, 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039876652999999998, 2024-04-20T02:53:07.735 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490276+0000", 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490785+0000", 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050941300000000001 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490804+0000", 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9080999999999999e-05 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490823+0000", 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.855e-05 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498138+0000", 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007315004 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498153+0000", 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5515e-05 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530153+0000", 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031999845999999998 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4016 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'146, mlcod=147'143)", 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.393957729, 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054600045999999999, 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262580+0000", 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262733+0000", 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015289700000000001 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262746+0000", 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.375e-05 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262753+0000", 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6159999999999998e-06 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262817+0000", 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4202999999999995e-05 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262842+0000", 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5296e-05 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317137+0000", 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054294664999999999 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317181+0000", 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4017 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'147, mlcod=147'143)", 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 19.378911042999999, 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039585437000000001, 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.743 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277626+0000", 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:07.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278058+0000", 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043228299999999999 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278074+0000", 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5775000000000001e-05 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278083+0000", 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2620000000000004e-06 2024-04-20T02:53:07.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286849+0000", 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087652210000000001 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286881+0000", 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1919999999999999e-05 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317188+0000", 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030307392999999998 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317213+0000", 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4953999999999999e-05 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4340 4.8 e163/114 4:1a8c7fb2:::smithi10548823-47:head v 163'162, mlcod=159'158)", 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 4.7442346579999999, 2024-04-20T02:53:07.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.04004311, 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912303+0000", 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912722+0000", 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00041911999999999999 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912738+0000", 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6294e-05 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912748+0000", 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6059999999999992e-06 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912840+0000", 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2143000000000002e-05 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912867+0000", 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7121e-05 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952298+0000", 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039430999000000001 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952347+0000", 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8990999999999999e-05 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:07.751 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:07.753 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:07.753 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:07.753+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:07.853 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:07.853+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:07.901 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:53:07.922 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:07.922 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:07.922 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:07.923 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:07.932 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:07.954 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:07.954+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:08.055 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:08.055+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:08.059 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:08.062 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:08.062 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:08.062 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:08.062 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:08.062 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:08.072 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:08.078 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:08.097 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:08.116 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:08.135 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:08.155 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:08.156 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:08.157+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:08.174 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:08.197 INFO:tasks.rados.rados.0.smithi105.stdout:1892: done (0 left) 2024-04-20T02:53:08.197 INFO:tasks.rados.rados.0.smithi105.stdout:1893: tier_evict oid 38 2024-04-20T02:53:08.197 INFO:tasks.rados.rados.0.smithi105.stdout:1893: tier_evict oid 38 snap -1 2024-04-20T02:53:08.197 INFO:tasks.rados.rados.0.smithi105.stdout:1894: read oid 47 snap -1 2024-04-20T02:53:08.197 INFO:tasks.rados.rados.0.smithi105.stdout:1894: expect (ObjNum 378 snap 40 seq_num 378) 2024-04-20T02:53:08.197 INFO:tasks.rados.rados.0.smithi105.stdout:1895: read oid 50 snap -1 2024-04-20T02:53:08.197 INFO:tasks.rados.rados.0.smithi105.stdout:1895: expect (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:53:08.197 INFO:tasks.rados.rados.0.smithi105.stdout:1896: read oid 16 snap -1 2024-04-20T02:53:08.197 INFO:tasks.rados.rados.0.smithi105.stdout:1896: expect deleted 2024-04-20T02:53:08.197 INFO:tasks.rados.rados.0.smithi105.stdout:1897: delete oid 12 current snap is 43 2024-04-20T02:53:08.198 INFO:tasks.rados.rados.0.smithi105.stdout:1893: got (22) Invalid argument 2024-04-20T02:53:08.203 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:08.205 INFO:tasks.rados.rados.0.smithi105.stdout:1894: expect (ObjNum 378 snap 40 seq_num 378) 2024-04-20T02:53:08.206 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:08.206 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:08.206 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:08.206 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:08.206 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:08.206 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:08.207 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:08.207 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:08.249 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:08.249 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:08.256 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:08.257+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:08.260 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:08.342 INFO:tasks.rados.rados.0.smithi105.stdout:1893: done (4 left) 2024-04-20T02:53:08.342 INFO:tasks.rados.rados.0.smithi105.stdout:1894: done (3 left) 2024-04-20T02:53:08.342 INFO:tasks.rados.rados.0.smithi105.stdout:1897: done (2 left) 2024-04-20T02:53:08.342 INFO:tasks.rados.rados.0.smithi105.stdout:1898: read oid 11 snap -1 2024-04-20T02:53:08.342 INFO:tasks.rados.rados.0.smithi105.stdout:1898: expect (ObjNum 347 snap 37 seq_num 347) 2024-04-20T02:53:08.342 INFO:tasks.rados.rados.0.smithi105.stdout:1899: read oid 47 snap -1 2024-04-20T02:53:08.342 INFO:tasks.rados.rados.0.smithi105.stdout:1899: expect (ObjNum 378 snap 40 seq_num 378) 2024-04-20T02:53:08.342 INFO:tasks.rados.rados.0.smithi105.stdout:1900: read oid 34 snap -1 2024-04-20T02:53:08.342 INFO:tasks.rados.rados.0.smithi105.stdout:1900: expect (ObjNum 377 snap 40 seq_num 377) 2024-04-20T02:53:08.342 INFO:tasks.rados.rados.0.smithi105.stdout:1901: delete oid 49 current snap is 43 2024-04-20T02:53:08.348 INFO:tasks.rados.rados.0.smithi105.stdout:1900: expect (ObjNum 377 snap 40 seq_num 377) 2024-04-20T02:53:08.357 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:08.358+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:08.395 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:08.395 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:08.405 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:08.459 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:08.459+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:08.507 INFO:tasks.rados.rados.0.smithi105.stdout:1898: expect (ObjNum 347 snap 37 seq_num 347) 2024-04-20T02:53:08.534 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:08.534 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:08.544 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:08.560 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:08.560+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:08.661 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:08.662+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:08.713 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:08.713 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:08.723 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:08.741 INFO:tasks.rados.rados.0.smithi105.stdout:1899: expect (ObjNum 378 snap 40 seq_num 378) 2024-04-20T02:53:08.762 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:08.762+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:08.853 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:08.853 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:08.864 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:08.864 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:08.864+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:08.871 INFO:tasks.rados.rados.0.smithi105.stdout:1898: done (5 left) 2024-04-20T02:53:08.872 INFO:tasks.rados.rados.0.smithi105.stdout:1899: done (4 left) 2024-04-20T02:53:08.872 INFO:tasks.rados.rados.0.smithi105.stdout:1900: done (3 left) 2024-04-20T02:53:08.872 INFO:tasks.rados.rados.0.smithi105.stdout:1901: done (2 left) 2024-04-20T02:53:08.872 INFO:tasks.rados.rados.0.smithi105.stdout:1902: read oid 22 snap 41 2024-04-20T02:53:08.872 INFO:tasks.rados.rados.0.smithi105.stdout:1902: expect (ObjNum 361 snap 37 seq_num 361) 2024-04-20T02:53:08.872 INFO:tasks.rados.rados.0.smithi105.stdout:1903: snap_create 2024-04-20T02:53:08.875 INFO:tasks.rados.rados.0.smithi105.stdout:1902: expect (ObjNum 361 snap 37 seq_num 361) 2024-04-20T02:53:08.965 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:08.965+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:09.033 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:09.033 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:09.044 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:09.084 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:09.085+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:09.166 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:09.166+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:09.201 INFO:tasks.rados.rados.0.smithi105.stdout:1902: done (3 left) 2024-04-20T02:53:09.201 INFO:tasks.rados.rados.0.smithi105.stdout:1903: done (2 left) 2024-04-20T02:53:09.201 INFO:tasks.rados.rados.0.smithi105.stdout:1904: write oid 28 current snap is 44 2024-04-20T02:53:09.201 INFO:tasks.rados.rados.0.smithi105.stdout:1904: seq_num 389 ranges {537923=405864,1513613=593796} 2024-04-20T02:53:09.211 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:53:09.211 INFO:tasks.rados.rados.0.smithi105.stdout:1904: writing smithi10548823-28 from 537923 to 943787 tid 1 2024-04-20T02:53:09.216 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:09.216 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:09.220 INFO:tasks.rados.rados.0.smithi105.stdout:1904: writing smithi10548823-28 from 1513613 to 2107409 tid 2 2024-04-20T02:53:09.220 INFO:tasks.rados.rados.0.smithi105.stdout:1905: write (excl) oid 44 current snap is 44 2024-04-20T02:53:09.220 INFO:tasks.rados.rados.0.smithi105.stdout:1905: seq_num 390 ranges {554730=481113,1572095=737454} 2024-04-20T02:53:09.221 INFO:tasks.rados.rados.0.smithi105.stdout:1905: writing smithi10548823-44 from 554730 to 1035843 tid 1 2024-04-20T02:53:09.229 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:09.229 INFO:tasks.rados.rados.0.smithi105.stdout:1905: writing smithi10548823-44 from 1572095 to 2309549 tid 2 2024-04-20T02:53:09.229 INFO:tasks.rados.rados.0.smithi105.stdout:1906: write oid 37 current snap is 44 2024-04-20T02:53:09.229 INFO:tasks.rados.rados.0.smithi105.stdout:1906: seq_num 391 ranges {571537=556362,1630577=597465} 2024-04-20T02:53:09.235 INFO:tasks.rados.rados.0.smithi105.stdout:1906: writing smithi10548823-37 from 571537 to 1127899 tid 1 2024-04-20T02:53:09.243 INFO:tasks.rados.rados.0.smithi105.stdout:1906: writing smithi10548823-37 from 1630577 to 2228042 tid 2 2024-04-20T02:53:09.243 INFO:tasks.rados.rados.0.smithi105.stdout:1907: read oid 41 snap 43 2024-04-20T02:53:09.243 INFO:tasks.rados.rados.0.smithi105.stdout:1907: expect (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:53:09.244 INFO:tasks.rados.rados.0.smithi105.stdout:1908: read oid 5 snap -1 2024-04-20T02:53:09.244 INFO:tasks.rados.rados.0.smithi105.stdout:1908: expect (ObjNum 368 snap 37 seq_num 368) 2024-04-20T02:53:09.244 INFO:tasks.rados.rados.0.smithi105.stdout:1909: read oid 26 snap -1 2024-04-20T02:53:09.244 INFO:tasks.rados.rados.0.smithi105.stdout:1909: expect (ObjNum 381 snap 41 seq_num 381) 2024-04-20T02:53:09.244 INFO:tasks.rados.rados.0.smithi105.stdout:1910: tier_flush oid 31 2024-04-20T02:53:09.244 INFO:tasks.rados.rados.0.smithi105.stdout:1910: tier_flush oid 31 snap -1 2024-04-20T02:53:09.244 INFO:tasks.rados.rados.0.smithi105.stdout:1911: write (excl) oid 4 current snap is 44 2024-04-20T02:53:09.252 INFO:tasks.rados.rados.0.smithi105.stdout:1911: seq_num 392 ranges {588344=631611,1689059=741123} 2024-04-20T02:53:09.260 INFO:tasks.rados.rados.0.smithi105.stdout:1911: writing smithi10548823-4 from 588344 to 1219955 tid 1 2024-04-20T02:53:09.264 INFO:tasks.rados.rados.0.smithi105.stdout:1911: writing smithi10548823-4 from 1689059 to 2430182 tid 2 2024-04-20T02:53:09.264 INFO:tasks.rados.rados.0.smithi105.stdout:1912: read oid 11 snap -1 2024-04-20T02:53:09.264 INFO:tasks.rados.rados.0.smithi105.stdout:1912: expect (ObjNum 347 snap 37 seq_num 347) 2024-04-20T02:53:09.264 INFO:tasks.rados.rados.0.smithi105.stdout:1913: read oid 25 snap -1 2024-04-20T02:53:09.264 INFO:tasks.rados.rados.0.smithi105.stdout:1913: expect (ObjNum 383 snap 41 seq_num 383) 2024-04-20T02:53:09.265 INFO:tasks.rados.rados.0.smithi105.stdout:1914: delete oid 6 current snap is 44 2024-04-20T02:53:09.277 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:09.277+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:09.277 INFO:tasks.rados.rados.0.smithi105.stdout:1895: expect (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:53:09.368 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:09.368+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:09.468 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:09.469+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:09.487 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:09.569 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:09.570+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:09.617 INFO:tasks.rados.rados.0.smithi105.stdout:1904: finishing write tid 1 to smithi10548823-28 2024-04-20T02:53:09.618 INFO:tasks.rados.rados.0.smithi105.stdout:1904: finishing write tid 2 to smithi10548823-28 2024-04-20T02:53:09.618 INFO:tasks.rados.rados.0.smithi105.stdout:1904: finishing write tid 3 to smithi10548823-28 2024-04-20T02:53:09.618 INFO:tasks.rados.rados.0.smithi105.stdout:1904: finishing write tid 4 to smithi10548823-28 2024-04-20T02:53:09.618 INFO:tasks.rados.rados.0.smithi105.stdout:1904: oid 28 updating version 0 to 215 2024-04-20T02:53:09.618 INFO:tasks.rados.rados.0.smithi105.stdout:1904: oid 28 version 215 is already newer than 214 2024-04-20T02:53:09.618 INFO:tasks.rados.rados.0.smithi105.stdout:1904: oid 28 updating version 215 to 216 2024-04-20T02:53:09.618 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 216 (ObjNum 389 snap 44 seq_num 389) dirty exists 2024-04-20T02:53:09.618 INFO:tasks.rados.rados.0.smithi105.stdout:1904: left oid 28 (ObjNum 389 snap 44 seq_num 389) 2024-04-20T02:53:09.618 INFO:tasks.rados.rados.0.smithi105.stdout:1907: expect (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:53:09.672 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:09.671+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:09.757 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:09.757 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:09.771 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:09.772 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:09.772+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:09.784 INFO:tasks.rados.rados.0.smithi105.stdout:1912: expect (ObjNum 347 snap 37 seq_num 347) 2024-04-20T02:53:09.872 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:09.872+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:09.974 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:09.974+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:09.980 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:09.980 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:09.991 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:10.059 INFO:tasks.rados.rados.0.smithi105.stdout:1908: expect (ObjNum 368 snap 37 seq_num 368) 2024-04-20T02:53:10.074 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:10.074+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:10.169 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:10.169 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:10.175 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:10.176+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:10.179 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:10.211 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:53:10.219 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:10.249 INFO:tasks.rados.rados.0.smithi105.stdout:1909: expect (ObjNum 381 snap 41 seq_num 381) 2024-04-20T02:53:10.292 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:10.292+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:10.346 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:10.346 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:10.357 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:10.358 INFO:tasks.rados.rados.0.smithi105.stdout:1905: finishing write tid 1 to smithi10548823-44 2024-04-20T02:53:10.358 INFO:tasks.rados.rados.0.smithi105.stdout:1905: finishing write tid 2 to smithi10548823-44 2024-04-20T02:53:10.358 INFO:tasks.rados.rados.0.smithi105.stdout:1905: finishing write tid 3 to smithi10548823-44 2024-04-20T02:53:10.358 INFO:tasks.rados.rados.0.smithi105.stdout:1895: done (12 left) 2024-04-20T02:53:10.358 INFO:tasks.rados.rados.0.smithi105.stdout:1896: done (11 left) 2024-04-20T02:53:10.358 INFO:tasks.rados.rados.0.smithi105.stdout:1904: done (10 left) 2024-04-20T02:53:10.358 INFO:tasks.rados.rados.0.smithi105.stdout:1907: done (9 left) 2024-04-20T02:53:10.358 INFO:tasks.rados.rados.0.smithi105.stdout:1908: done (8 left) 2024-04-20T02:53:10.358 INFO:tasks.rados.rados.0.smithi105.stdout:1909: done (7 left) 2024-04-20T02:53:10.359 INFO:tasks.rados.rados.0.smithi105.stdout:1912: done (6 left) 2024-04-20T02:53:10.359 INFO:tasks.rados.rados.0.smithi105.stdout:1914: done (5 left) 2024-04-20T02:53:10.359 INFO:tasks.rados.rados.0.smithi105.stdout:1915: set_chunk oid 40 target oid 2024-04-20T02:53:10.401 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:10.401 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:10.401+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:10.420 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:10.439 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:10.458 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:10.478 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:10.478 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:10.479+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:10.494 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 143.90868650900001, 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.495 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:53:10.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:53:10.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:53:10.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:53:10.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:53:10.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:10.501 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 143.88795719800001, 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:10.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:53:10.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:53:10.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.507 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 143.87870544699999, 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.508 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:53:10.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:53:10.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:53:10.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 136.081706231, 2024-04-20T02:53:10.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:53:10.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.518 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 136.07570144300001, 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.522 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4029 4.13 4:cf11ccc0:::smithi10548823-4:head [write 699257~564637,stat] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.938158927, 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.86467550400000004, 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4029 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555666+0000", 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555809+0000", 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014306900000000001 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555832+0000", 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3680999999999999e-05 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555846+0000", 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3654e-05 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555913+0000", 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6736999999999995e-05 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556058+0000", 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014538100000000001 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556392+0000", 2024-04-20T02:53:10.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033393099999999998 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578661+0000", 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022268553999999999 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578702+0000", 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1533999999999999e-05 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578747+0000", 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4858999999999999e-05 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578811+0000", 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3520000000000002e-05 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605743+0000", 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026932418999999999 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420312+0000", 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81456849600000003 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420343+0000", 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1911e-05 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4030 4.13 4:cf11ccc0:::smithi10548823-4:head [write 1873062~640438] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:10.529 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.931245806, 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85780235299999996, 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4030 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562579+0000", 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562714+0000", 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013475 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562728+0000", 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3516000000000001e-05 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562736+0000", 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5690000000000005e-06 2024-04-20T02:53:10.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578835+0000", 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016099156 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578957+0000", 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121094 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.579123+0000", 2024-04-20T02:53:10.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016647100000000001 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605727+0000", 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026603643999999999 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605749+0000", 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2070999999999999e-05 2024-04-20T02:53:10.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605799+0000", 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0417000000000002e-05 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605841+0000", 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2344e-05 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605907+0000", 2024-04-20T02:53:10.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5135999999999994e-05 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420373+0000", 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81446675899999998 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420383+0000", 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0108999999999999e-05 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4031 4.13 4:cf11ccc0:::smithi10548823-4:head [write 3305941~593316] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.923684230999999, 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85027709799999995, 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4031 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570142+0000", 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570277+0000", 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135606 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570288+0000", 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0719e-05 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570294+0000", 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2519999999999996e-06 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592786+0000", 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022491220999999999 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592911+0000", 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012546599999999999 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.593118+0000", 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000207011 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605878+0000", 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012760298999999999 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605934+0000", 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5813999999999998e-05 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605963+0000", 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9275e-05 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.542 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606612+0000", 2024-04-20T02:53:10.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00064813999999999996 2024-04-20T02:53:10.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.542 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613782+0000", 2024-04-20T02:53:10.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071705670000000001 2024-04-20T02:53:10.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420411+0000", 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.80662907399999995 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420420+0000", 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5790000000000004e-06 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4032 4.13 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (56) in=7b,truncate 3899257] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:10.543 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.923527802999999, 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85015831100000006, 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4032 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570302+0000", 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1700000000000001e-06 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570305+0000", 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.976e-06 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570308+0000", 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8899999999999999e-06 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605985+0000", 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035677121999999999 2024-04-20T02:53:10.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606066+0000", 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0797000000000006e-05 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606211+0000", 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014569999999999999 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607013+0000", 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00080160599999999995 2024-04-20T02:53:10.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607046+0000", 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2938999999999998e-05 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607086+0000", 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0203000000000001e-05 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607157+0000", 2024-04-20T02:53:10.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0621e-05 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613833+0000", 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066762269999999999 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420448+0000", 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.806615478 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420457+0000", 2024-04-20T02:53:10.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9779999999999994e-06 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4033 4.13 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.923515929000001, 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85021192700000003, 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.550 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4033 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570312+0000", 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419999999999999e-06 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570316+0000", 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3730000000000001e-06 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570318+0000", 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6680000000000001e-06 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606541+0000", 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036222797000000001 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606598+0000", 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6629999999999998e-05 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420515+0000", 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81391740099999998 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420523+0000", 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6799999999999993e-06 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4034 4.13 4:cfd1e4d4:::smithi10548823-9:head [rollback 3f] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:10.554 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.923506290999999, 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85017147900000001, 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4034 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.555 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570320+0000", 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570322+0000", 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.37e-06 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570325+0000", 2024-04-20T02:53:10.556 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7800000000000001e-06 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570328+0000", 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5379999999999998e-06 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606632+0000", 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036304075999999998 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606716+0000", 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3689000000000005e-05 2024-04-20T02:53:10.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420484+0000", 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81376812700000001 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420492+0000", 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1179999999999992e-06 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.558 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4062 4.9 4:9c87d36d:::smithi10548823-24:head [write 766485~465633] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.343652509999998, 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075029242999999995, 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4062 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:10.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150173+0000", 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150435+0000", 2024-04-20T02:53:10.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000262064 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150445+0000", 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8160000000000005e-06 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150450+0000", 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1880000000000001e-06 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150484+0000", 2024-04-20T02:53:10.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150490+0000", 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6810000000000003e-06 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161686+0000", 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011196187 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161840+0000", 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015389000000000001 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.162103+0000", 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026333299999999998 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224927+0000", 2024-04-20T02:53:10.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062823458999999998 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224993+0000", 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6667000000000004e-05 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225059+0000", 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6117000000000004e-05 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225119+0000", 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9342000000000002e-05 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225204+0000", 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4844000000000003e-05 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4063 4.9 4:9c87d36d:::smithi10548823-24:head [write 1990637~647776] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.343369836000001, 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075097435000000004, 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.565 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4063 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150746+0000", 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028951399999999999 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150753+0000", 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150757+0000", 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2300000000000002e-06 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150801+0000", 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3421000000000003e-05 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:10.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150811+0000", 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0581000000000001e-05 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177065+0000", 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02625367 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177164+0000", 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8900999999999994e-05 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:10.569 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177319+0000", 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000155437 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225048+0000", 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047728857 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225217+0000", 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168395 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225375+0000", 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157837 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225437+0000", 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2824999999999996e-05 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225555+0000", 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117169 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.571 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4064 4.9 4:9c87d36d:::smithi10548823-24:head [write 3299280~667205] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.343063083000001, 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112222372, 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4064 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150763+0000", 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151076+0000", 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000312355 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151083+0000", 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1010000000000003e-06 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151091+0000", 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0409999999999998e-06 2024-04-20T02:53:10.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151127+0000", 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151133+0000", 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1099999999999999e-06 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200197+0000", 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049064000000000003 2024-04-20T02:53:10.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200315+0000", 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117855 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200545+0000", 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022990700000000001 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225595+0000", 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025050224999999999 2024-04-20T02:53:10.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262670+0000", 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075460999999997 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262767+0000", 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6479999999999998e-05 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262835+0000", 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999999e-05 2024-04-20T02:53:10.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262986+0000", 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015129799999999999 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4065 4.9 4:9c87d36d:::smithi10548823-24:head [setxattr _header (57) in=7b,truncate 3966485] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.342723202999998, 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112193353, 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.578 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4065 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151103+0000", 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151107+0000", 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151111+0000", 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2309999999999998e-06 2024-04-20T02:53:10.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151116+0000", 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4850000000000003e-06 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151168+0000", 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2673000000000002e-05 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151179+0000", 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0112000000000001e-05 2024-04-20T02:53:10.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225611+0000", 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074432549000000001 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225689+0000", 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7612000000000002e-05 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225858+0000", 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016945400000000001 2024-04-20T02:53:10.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262650+0000", 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036792126000000001 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262678+0000", 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8132000000000001e-05 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263077+0000", 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039868299999999999 2024-04-20T02:53:10.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263141+0000", 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3448999999999995e-05 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263297+0000", 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015649699999999999 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.584 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4066 4.9 4:9c87d36d:::smithi10548823-24:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.342706347, 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112663448, 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4066 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.585 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151120+0000", 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.586 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151123+0000", 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4389999999999999e-06 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151128+0000", 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2449999999999998e-06 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151132+0000", 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9049999999999999e-06 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.587 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151198+0000", 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5988000000000006e-05 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151205+0000", 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8770000000000004e-06 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226171+0000", 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074966844000000005 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:10.588 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226261+0000", 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9975999999999994e-05 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263337+0000", 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075863000000001 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263443+0000", 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000105678 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.589 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263784+0000", 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034103200000000001 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4095 4.9 4:964390c0:::smithi10548823-36:head [write 450520~558231] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.342692277000001, 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112875926, 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.590 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4095 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.591 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151134+0000", 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151379+0000", 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024486000000000001 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151385+0000", 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1460000000000001e-06 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.592 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151389+0000", 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6610000000000002e-06 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151413+0000", 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3611000000000001e-05 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151419+0000", 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8780000000000003e-06 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226302+0000", 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074883540999999998 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226460+0000", 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015792000000000001 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226599+0000", 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013939599999999999 2024-04-20T02:53:10.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262685+0000", 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036086115000000002 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262726+0000", 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0448999999999998e-05 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263833+0000", 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001106685 2024-04-20T02:53:10.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263884+0000", 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1530999999999998e-05 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264011+0000", 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012641900000000001 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.596 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4096 4.9 4:964390c0:::smithi10548823-36:head [write 1483047~515125] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.342432742, 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11703358799999999, 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4096 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.597 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.598 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151644+0000", 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025029799999999998 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151651+0000", 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.148e-06 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151655+0000", 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2769999999999999e-06 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.599 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151689+0000", 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711999999999998e-05 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151698+0000", 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0359999999999995e-06 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245111+0000", 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093413164000000007 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245232+0000", 2024-04-20T02:53:10.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000120214 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245424+0000", 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019272500000000001 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263033+0000", 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017608461999999998 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266386+0000", 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033530980000000001 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268218+0000", 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001832101 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268283+0000", 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5334999999999997e-05 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268428+0000", 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014449999999999999 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4097 4.9 4:964390c0:::smithi10548823-36:head [setxattr _header (57) in=7b,truncate 2050520] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.342164789000002, 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.117031099, 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.603 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4097 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151665+0000", 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.614e-06 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151670+0000", 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:53:10.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151675+0000", 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0560000000000004e-06 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151738+0000", 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2447999999999996e-05 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151753+0000", 2024-04-20T02:53:10.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5007e-05 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264044+0000", 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112291562 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264165+0000", 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121081 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264421+0000", 2024-04-20T02:53:10.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025568599999999998 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266133+0000", 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001711668 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266755+0000", 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00062224100000000005 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268484+0000", 2024-04-20T02:53:10.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001729169 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268545+0000", 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0365000000000002e-05 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268693+0000", 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014860000000000001 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4098 4.9 4:964390c0:::smithi10548823-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.342147307000001, 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11735205, 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4098 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:10.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151679+0000", 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:10.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151681+0000", 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3140000000000002e-06 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151686+0000", 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0209999999999999e-06 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151691+0000", 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4700000000000004e-06 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151779+0000", 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8399999999999994e-05 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151788+0000", 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7129999999999994e-06 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265294+0000", 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11350631899999999 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265391+0000", 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6861000000000007e-05 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268730+0000", 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003339102 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268841+0000", 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110708 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:10.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:10.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.269032+0000", 2024-04-20T02:53:10.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019057899999999999 2024-04-20T02:53:10.615 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.615 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.615 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.615 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:10.615 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:10.615 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:10.616 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:10.617 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:10.617 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:10.617 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:10.618 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:10.618 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:10.618 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:10.618 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:10.619 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:10.617+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:10.717 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:10.718+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:10.729 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 3, 6, 7, 5, 2, 4] 2024-04-20T02:53:10.729 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:53:10.729 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-04-20T02:53:10.729 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-04-20T02:53:10.729 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:53:10.790 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:10.791 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:10.791 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:10.791 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:10.802 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:10.818 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:10.818+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:10.918 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:10.918+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:10.933 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:10.933 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:10.934 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:10.934 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:10.934 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:10.944 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:11.028 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:11.028+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:11.079 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:53:11.080 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:53:11.091 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-04-20T02:53:10.204155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":301021434,"num_objects":206,"num_object_clones":44,"num_object_copies":618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":16,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":4,"num_read":3513,"num_read_kb":3247925,"num_write":2215,"num_write_kb":576705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":158209138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":35,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":102,"acting":99,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":742236,"kb_used_data":534328,"kb_used_omap":9,"kb_used_meta":207798,"kb_avail":565488804,"statfs":{"total":579820584960,"available":579060535296,"internally_reserved":0,"allocated":547151872,"data_stored":610882745,"data_compressed":10650373,"data_compressed_allocated":11431936,"data_compressed_original":27213824,"omap_allocated":9619,"internal_metadata":212785773},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5161278,"num_objects":16,"num_object_clones":7,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-36,"num_objects_misplaced":-3,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":139,"num_read_kb":153659,"num_write":46,"num_write_kb":8402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":11390359,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.026111"},"pg_stats":[{"pgid":"4.1","version":"157'133","reported_seq":619,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-20T02:53:07.192428+0000","last_change":"2024-04-20T02:53:06.183706+0000","last_active":"2024-04-20T02:53:07.192428+0000","last_peered":"2024-04-20T02:53:07.192428+0000","last_clean":"2024-04-20T02:53:07.192428+0000","last_became_active":"2024-04-20T02:52:54.150080+0000","last_became_peered":"2024-04-20T02:52:54.150080+0000","last_unstale":"2024-04-20T02:53:07.192428+0000","last_undegraded":"2024-04-20T02:53:07.192428+0000","last_fullsized":"2024-04-20T02:53:07.192428+0000","mapping_epoch":154,"log_start":"137'127","ondisk_log_start":"137'127","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'131","last_scrub_stamp":"2024-04-20T02:52:54.612743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.612743+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":123,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:18.713556+0000","scrub_duration":0.0022862500000000001,"objects_trimmed":0,"snaptrim_duration":0.0026097640000000001,"stat_sum":{"num_bytes":3226977,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":107162,"num_write":18,"num_write_kb":5336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3980303,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":81,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-20T02:53:08.234186+0000","last_change":"2024-04-20T02:52:22.267305+0000","last_active":"2024-04-20T02:53:08.234186+0000","last_peered":"2024-04-20T02:53:08.234186+0000","last_clean":"2024-04-20T02:53:08.234186+0000","last_became_active":"2024-04-20T02:52:22.267070+0000","last_became_peered":"2024-04-20T02:52:22.267070+0000","last_unstale":"2024-04-20T02:53:08.234186+0000","last_undegraded":"2024-04-20T02:53:08.234186+0000","last_fullsized":"2024-04-20T02:53:08.234186+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:56.983760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.983760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:20.279288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"165'177","reported_seq":618,"reported_epoch":167,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:53:06.413995+0000","last_change":"2024-04-20T02:52:44.409412+0000","last_active":"2024-04-20T02:53:06.413995+0000","last_peered":"2024-04-20T02:53:06.413995+0000","last_clean":"2024-04-20T02:52:41.821209+0000","last_became_active":"2024-04-20T02:52:44.357388+0000","last_became_peered":"2024-04-20T02:52:44.357388+0000","last_unstale":"2024-04-20T02:53:06.413995+0000","last_undegraded":"2024-04-20T02:53:06.413995+0000","last_fullsized":"2024-04-20T02:53:06.413995+0000","mapping_epoch":145,"log_start":"163'173","ondisk_log_start":"163'173","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":169,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001820667,"objects_trimmed":0,"snaptrim_duration":0.00036920600000000002,"stat_sum":{"num_bytes":15409974,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":170,"num_read_kb":202173,"num_write":72,"num_write_kb":21310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"1"},{"start":"3a","length":"1"},{"start":"43","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":181,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-04-20T02:53:06.285464+0000","last_change":"2024-04-20T02:52:51.154350+0000","last_active":"2024-04-20T02:53:06.285464+0000","last_peered":"2024-04-20T02:53:06.285464+0000","last_clean":"2024-04-20T02:53:06.285464+0000","last_became_active":"2024-04-20T02:52:51.153987+0000","last_became_peered":"2024-04-20T02:52:51.153987+0000","last_unstale":"2024-04-20T02:53:06.285464+0000","last_undegraded":"2024-04-20T02:53:06.285464+0000","last_fullsized":"2024-04-20T02:53:06.285464+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:57.514743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:57.514743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:09.129491+0000","scrub_duration":0.001500723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":193,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-20T02:53:08.234033+0000","last_change":"2024-04-20T02:52:53.937048+0000","last_active":"2024-04-20T02:53:08.234033+0000","last_peered":"2024-04-20T02:53:08.234033+0000","last_clean":"2024-04-20T02:53:08.234033+0000","last_became_active":"2024-04-20T02:52:51.154174+0000","last_became_peered":"2024-04-20T02:52:51.154174+0000","last_unstale":"2024-04-20T02:53:08.234033+0000","last_undegraded":"2024-04-20T02:53:08.234033+0000","last_fullsized":"2024-04-20T02:53:08.234033+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:53.936957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:53.936957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.656218+0000","scrub_duration":0.0013534090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"163'248","reported_seq":750,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-04-20T02:53:09.256692+0000","last_change":"2024-04-20T02:53:06.180203+0000","last_active":"2024-04-20T02:53:09.256692+0000","last_peered":"2024-04-20T02:53:09.256692+0000","last_clean":"2024-04-20T02:53:09.256692+0000","last_became_active":"2024-04-20T02:51:18.502004+0000","last_became_peered":"2024-04-20T02:51:18.502004+0000","last_unstale":"2024-04-20T02:53:09.256692+0000","last_undegraded":"2024-04-20T02:53:09.256692+0000","last_fullsized":"2024-04-20T02:53:09.256692+0000","mapping_epoch":47,"log_start":"157'244","ondisk_log_start":"157'244","created":22,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'243","last_scrub_stamp":"2024-04-20T02:52:54.597931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.597931+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":240,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:20.264927+0000","scrub_duration":0.0028458709999999998,"objects_trimmed":0,"snaptrim_duration":0.000722724,"stat_sum":{"num_bytes":5630444,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":184904,"num_write":34,"num_write_kb":10563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"3.1","version":"170'243","reported_seq":559,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-04-20T02:53:09.334715+0000","last_change":"2024-04-20T02:53:08.766997+0000","last_active":"2024-04-20T02:53:09.334715+0000","last_peered":"2024-04-20T02:53:09.334715+0000","last_clean":"2024-04-20T02:53:09.334715+0000","last_became_active":"2024-04-20T02:52:52.148432+0000","last_became_peered":"2024-04-20T02:52:52.148432+0000","last_unstale":"2024-04-20T02:53:09.334715+0000","last_undegraded":"2024-04-20T02:53:09.334715+0000","last_fullsized":"2024-04-20T02:53:09.334715+0000","mapping_epoch":152,"log_start":"163'239","ondisk_log_start":"163'239","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":4,"log_dups_size":264,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:29.132762+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28268925,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":217,"num_read_kb":8220,"num_write":291,"num_write_kb":75525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-20T02:53:08.233867+0000","last_change":"2024-04-20T02:52:32.918930+0000","last_active":"2024-04-20T02:53:08.233867+0000","last_peered":"2024-04-20T02:53:08.233867+0000","last_clean":"2024-04-20T02:53:08.233867+0000","last_became_active":"2024-04-20T02:52:22.261878+0000","last_became_peered":"2024-04-20T02:52:22.261878+0000","last_unstale":"2024-04-20T02:53:08.233867+0000","last_undegraded":"2024-04-20T02:53:08.233867+0000","last_fullsized":"2024-04-20T02:53:08.233867+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:32.918813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.918813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:36.236915+0000","scrub_duration":0.0014288059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"162'133","reported_seq":569,"reported_epoch":167,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:53:06.287433+0000","last_change":"2024-04-20T02:52:44.373628+0000","last_active":"2024-04-20T02:53:06.287433+0000","last_peered":"2024-04-20T02:53:06.287433+0000","last_clean":"2024-04-20T02:52:39.159207+0000","last_became_active":"2024-04-20T02:52:44.361391+0000","last_became_peered":"2024-04-20T02:52:44.361391+0000","last_unstale":"2024-04-20T02:53:06.287433+0000","last_undegraded":"2024-04-20T02:53:06.287433+0000","last_fullsized":"2024-04-20T02:53:06.287433+0000","mapping_epoch":145,"log_start":"159'130","ondisk_log_start":"159'130","created":22,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'57","last_scrub_stamp":"2024-04-20T02:51:21.386796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:21.386796+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":217,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018878359999999999,"objects_trimmed":0,"snaptrim_duration":0.010828997,"stat_sum":{"num_bytes":17065979,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":132,"num_read_kb":152538,"num_write":37,"num_write_kb":9540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7937542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,0],"avail_no_missing":["1","0","7"],"object_location_counts":[{"shards":"0,1,7","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"1"},{"start":"3a","length":"1"},{"start":"43","length":"1"}]},{"pgid":"3.2","version":"163'252","reported_seq":533,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-04-20T02:53:06.414786+0000","last_change":"2024-04-20T02:51:44.446457+0000","last_active":"2024-04-20T02:53:06.414786+0000","last_peered":"2024-04-20T02:53:06.414786+0000","last_clean":"2024-04-20T02:53:06.414786+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:53:06.414786+0000","last_undegraded":"2024-04-20T02:53:06.414786+0000","last_fullsized":"2024-04-20T02:53:06.414786+0000","mapping_epoch":49,"log_start":"163'243","ondisk_log_start":"163'243","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'130","last_scrub_stamp":"2024-04-20T02:51:44.446332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:51:44.446332+0000","objects_scrubbed":20,"log_size":9,"log_dups_size":251,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:11.809200+0000","scrub_duration":0.0039636890000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24029897,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":224,"num_read_kb":13454,"num_write":296,"num_write_kb":87895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":336,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-20T02:53:08.234166+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:53:08.234166+0000","last_peered":"2024-04-20T02:53:08.234166+0000","last_clean":"2024-04-20T02:53:08.234166+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:53:08.234166+0000","last_undegraded":"2024-04-20T02:53:08.234166+0000","last_fullsized":"2024-04-20T02:53:08.234166+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":169,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-20T02:53:08.364116+0000","last_change":"2024-04-20T02:53:08.364116+0000","last_active":"2024-04-20T02:53:08.364116+0000","last_peered":"2024-04-20T02:53:08.364116+0000","last_clean":"2024-04-20T02:53:08.364116+0000","last_became_active":"2024-04-20T02:52:22.265487+0000","last_became_peered":"2024-04-20T02:52:22.265487+0000","last_unstale":"2024-04-20T02:53:08.364116+0000","last_undegraded":"2024-04-20T02:53:08.364116+0000","last_fullsized":"2024-04-20T02:53:08.364116+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.792324+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"161'96","reported_seq":339,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-20T02:53:08.234169+0000","last_change":"2024-04-20T02:53:06.189691+0000","last_active":"2024-04-20T02:53:08.234169+0000","last_peered":"2024-04-20T02:53:08.234169+0000","last_clean":"2024-04-20T02:53:08.234169+0000","last_became_active":"2024-04-20T02:52:54.150076+0000","last_became_peered":"2024-04-20T02:52:54.150076+0000","last_unstale":"2024-04-20T02:53:08.234169+0000","last_undegraded":"2024-04-20T02:53:08.234169+0000","last_fullsized":"2024-04-20T02:53:08.234169+0000","mapping_epoch":154,"log_start":"148'93","ondisk_log_start":"148'93","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'86","last_scrub_stamp":"2024-04-20T02:52:32.901998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.901998+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":123,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:38.125305+0000","scrub_duration":0.0024332220000000001,"objects_trimmed":0,"snaptrim_duration":0.00066730900000000004,"stat_sum":{"num_bytes":6050619,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":32313,"num_write":10,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1543958,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"3.0","version":"165'208","reported_seq":465,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-04-20T02:53:06.287284+0000","last_change":"2024-04-20T02:52:42.201016+0000","last_active":"2024-04-20T02:53:06.287284+0000","last_peered":"2024-04-20T02:53:06.287284+0000","last_clean":"2024-04-20T02:53:06.287284+0000","last_became_active":"2024-04-20T02:52:25.272853+0000","last_became_peered":"2024-04-20T02:52:25.272853+0000","last_unstale":"2024-04-20T02:53:06.287284+0000","last_undegraded":"2024-04-20T02:53:06.287284+0000","last_fullsized":"2024-04-20T02:53:06.287284+0000","mapping_epoch":126,"log_start":"161'205","ondisk_log_start":"161'205","created":21,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'188","last_scrub_stamp":"2024-04-20T02:52:42.200865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:52:42.200865+0000","objects_scrubbed":31,"log_size":3,"log_dups_size":217,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.320740+0000","scrub_duration":0.0052191850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27533891,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":170,"num_read_kb":5858,"num_write":248,"num_write_kb":72053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":15696807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-04-20T02:53:06.426935+0000","last_change":"2024-04-20T02:52:56.212555+0000","last_active":"2024-04-20T02:53:06.426935+0000","last_peered":"2024-04-20T02:53:06.426935+0000","last_clean":"2024-04-20T02:53:06.426935+0000","last_became_active":"2024-04-20T02:52:22.272822+0000","last_became_peered":"2024-04-20T02:52:22.272822+0000","last_unstale":"2024-04-20T02:53:06.426935+0000","last_undegraded":"2024-04-20T02:53:06.426935+0000","last_fullsized":"2024-04-20T02:53:06.426935+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:56.212430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.212430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:20.820102+0000","scrub_duration":0.0013799870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"163'123","reported_seq":654,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-20T02:53:07.192514+0000","last_change":"2024-04-20T02:53:06.184075+0000","last_active":"2024-04-20T02:53:07.192514+0000","last_peered":"2024-04-20T02:53:07.192514+0000","last_clean":"2024-04-20T02:53:07.192514+0000","last_became_active":"2024-04-20T02:52:46.141983+0000","last_became_peered":"2024-04-20T02:52:46.141983+0000","last_unstale":"2024-04-20T02:53:07.192514+0000","last_undegraded":"2024-04-20T02:53:07.192514+0000","last_fullsized":"2024-04-20T02:53:07.192514+0000","mapping_epoch":147,"log_start":"157'119","ondisk_log_start":"157'119","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'119","last_scrub_stamp":"2024-04-20T02:52:56.564632+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.564632+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":305,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:22.552034+0000","scrub_duration":0.0028929759999999998,"objects_trimmed":0,"snaptrim_duration":0.0025310549999999999,"stat_sum":{"num_bytes":3613538,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":29933,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"3.3","version":"170'215","reported_seq":442,"reported_epoch":170,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-04-20T02:53:09.557936+0000","last_change":"2024-04-20T02:53:09.208357+0000","last_active":"2024-04-20T02:53:09.557936+0000","last_peered":"2024-04-20T02:53:09.557936+0000","last_clean":"2024-04-20T02:52:46.141075+0000","last_became_active":"2024-04-20T02:53:09.205680+0000","last_became_peered":"2024-04-20T02:53:09.205680+0000","last_unstale":"2024-04-20T02:53:09.557936+0000","last_undegraded":"2024-04-20T02:53:09.201002+0000","last_fullsized":"2024-04-20T02:53:09.200904+0000","mapping_epoch":169,"log_start":"163'211","ondisk_log_start":"163'211","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'165","last_scrub_stamp":"2024-04-20T02:52:05.855745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:52:05.855745+0000","objects_scrubbed":23,"log_size":4,"log_dups_size":217,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:12.818299+0000","scrub_duration":0.0059089679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25591970,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":186,"num_read_kb":6604,"num_write":260,"num_write_kb":71433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4888846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6],"avail_no_missing":["0","4","6"],"object_location_counts":[{"shards":"0,6","objects":28}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":148,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-20T02:53:08.201982+0000","last_change":"2024-04-20T02:51:52.777359+0000","last_active":"2024-04-20T02:53:08.201982+0000","last_peered":"2024-04-20T02:53:08.201982+0000","last_clean":"2024-04-20T02:53:08.201982+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:53:08.201982+0000","last_undegraded":"2024-04-20T02:53:08.201982+0000","last_fullsized":"2024-04-20T02:53:08.201982+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:52.777234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:52.777234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:10.584062+0000","scrub_duration":0.0012962270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"56'17","reported_seq":428,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-04-20T02:53:06.288933+0000","last_change":"2024-04-20T02:53:06.288933+0000","last_active":"2024-04-20T02:53:06.288933+0000","last_peered":"2024-04-20T02:53:06.288933+0000","last_clean":"2024-04-20T02:53:06.288933+0000","last_became_active":"2024-04-20T02:52:40.140298+0000","last_became_peered":"2024-04-20T02:52:40.140298+0000","last_unstale":"2024-04-20T02:53:06.288933+0000","last_undegraded":"2024-04-20T02:53:06.288933+0000","last_fullsized":"2024-04-20T02:53:06.288933+0000","mapping_epoch":141,"log_start":"53'14","ondisk_log_start":"53'14","created":22,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:52:56.000178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.000178+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:03.483771+0000","scrub_duration":0.0015078069999999999,"objects_trimmed":0,"snaptrim_duration":0.00095689699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"4.2","version":"0'0","reported_seq":192,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-20T02:53:07.192470+0000","last_change":"2024-04-20T02:53:06.183128+0000","last_active":"2024-04-20T02:53:07.192470+0000","last_peered":"2024-04-20T02:53:07.192470+0000","last_clean":"2024-04-20T02:53:07.192470+0000","last_became_active":"2024-04-20T02:52:51.150057+0000","last_became_peered":"2024-04-20T02:52:51.150057+0000","last_unstale":"2024-04-20T02:53:07.192470+0000","last_undegraded":"2024-04-20T02:53:07.192470+0000","last_fullsized":"2024-04-20T02:53:07.192470+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_clean_scrub_stamp":"2024-04-20T02:52:43.686760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:01.323681+0000","scrub_duration":0.0018670150000000001,"objects_trimmed":0,"snaptrim_duration":0.0022521170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":172,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-04-20T02:53:06.287840+0000","last_change":"2024-04-20T02:52:18.363613+0000","last_active":"2024-04-20T02:53:06.287840+0000","last_peered":"2024-04-20T02:53:06.287840+0000","last_clean":"2024-04-20T02:53:06.287840+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:53:06.287840+0000","last_undegraded":"2024-04-20T02:53:06.287840+0000","last_fullsized":"2024-04-20T02:53:06.287840+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:18.363433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:18.363433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:38.745528+0000","scrub_duration":0.0016555929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"167'166","reported_seq":605,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-04-20T02:53:08.345108+0000","last_change":"2024-04-20T02:53:06.184590+0000","last_active":"2024-04-20T02:53:08.345108+0000","last_peered":"2024-04-20T02:53:08.345108+0000","last_clean":"2024-04-20T02:53:08.345108+0000","last_became_active":"2024-04-20T02:52:13.145577+0000","last_became_peered":"2024-04-20T02:52:13.145577+0000","last_unstale":"2024-04-20T02:53:08.345108+0000","last_undegraded":"2024-04-20T02:53:08.345108+0000","last_fullsized":"2024-04-20T02:53:08.345108+0000","mapping_epoch":62,"log_start":"163'160","ondisk_log_start":"163'160","created":22,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'74","last_scrub_stamp":"2024-04-20T02:51:56.784299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.784299+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":318,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:15.219512+0000","scrub_duration":0.0015920019999999999,"objects_trimmed":0,"snaptrim_duration":0.0042424669999999998,"stat_sum":{"num_bytes":10675116,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":63,"num_read_kb":63768,"num_write":41,"num_write_kb":10973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"4.9","version":"167'456","reported_seq":982,"reported_epoch":168,"state":"remapped+peering","last_fresh":"2024-04-20T02:53:07.195336+0000","last_change":"2024-04-20T02:53:07.195564+0000","last_active":"2024-04-20T02:53:07.195336+0000","last_peered":"2024-04-20T02:53:06.458011+0000","last_clean":"2024-04-20T02:52:47.958346+0000","last_became_active":"2024-04-20T02:52:52.160204+0000","last_became_peered":"2024-04-20T02:52:52.160204+0000","last_unstale":"2024-04-20T02:53:07.195336+0000","last_undegraded":"2024-04-20T02:53:07.195336+0000","last_fullsized":"2024-04-20T02:53:07.195336+0000","mapping_epoch":168,"log_start":"166'452","ondisk_log_start":"166'452","created":22,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'294","last_scrub_stamp":"2024-04-20T02:52:16.813424+0000","last_deep_scrub":"98'280","last_deep_scrub_stamp":"2024-04-20T02:51:56.947305+0000","last_clean_scrub_stamp":"2024-04-20T02:52:16.813424+0000","objects_scrubbed":7,"log_size":4,"log_dups_size":1073,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:39.239031+0000","scrub_duration":0.0034018329999999999,"objects_trimmed":1,"snaptrim_duration":0.075274541,"stat_sum":{"num_bytes":40305411,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":113,"num_read_kb":105534,"num_write":89,"num_write_kb":25893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"41","length":"1"},{"start":"43","length":"1"}]},{"pgid":"4.a","version":"165'108","reported_seq":508,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-04-20T02:53:06.288491+0000","last_change":"2024-04-20T02:53:06.288491+0000","last_active":"2024-04-20T02:53:06.288491+0000","last_peered":"2024-04-20T02:53:06.288491+0000","last_clean":"2024-04-20T02:53:06.288491+0000","last_became_active":"2024-04-20T02:52:10.138005+0000","last_became_peered":"2024-04-20T02:52:10.138005+0000","last_unstale":"2024-04-20T02:53:06.288491+0000","last_undegraded":"2024-04-20T02:53:06.288491+0000","last_fullsized":"2024-04-20T02:53:06.288491+0000","mapping_epoch":111,"log_start":"158'105","ondisk_log_start":"158'105","created":22,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'53","last_scrub_stamp":"2024-04-20T02:52:05.810627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:05.810627+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":105,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:13.232408+0000","scrub_duration":0.0022334680000000002,"objects_trimmed":0,"snaptrim_duration":0.001099334,"stat_sum":{"num_bytes":5579957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":205658,"num_write":56,"num_write_kb":14550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"4.b","version":"170'295","reported_seq":740,"reported_epoch":170,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-04-20T02:53:09.557745+0000","last_change":"2024-04-20T02:53:09.208205+0000","last_active":"2024-04-20T02:53:09.557745+0000","last_peered":"2024-04-20T02:53:09.557745+0000","last_clean":"2024-04-20T02:52:49.029405+0000","last_became_active":"2024-04-20T02:53:09.206819+0000","last_became_peered":"2024-04-20T02:53:09.206819+0000","last_unstale":"2024-04-20T02:53:09.557745+0000","last_undegraded":"2024-04-20T02:53:09.201054+0000","last_fullsized":"2024-04-20T02:53:09.200843+0000","mapping_epoch":169,"log_start":"165'291","ondisk_log_start":"165'291","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'157","last_scrub_stamp":"2024-04-20T02:52:07.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:07.013316+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":291,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:19.720231+0000","scrub_duration":0.002193338,"objects_trimmed":0,"snaptrim_duration":0.0055389219999999999,"stat_sum":{"num_bytes":23610080,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":347,"num_read_kb":425847,"num_write":160,"num_write_kb":44102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7050982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":13}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"41","length":"1"},{"start":"43","length":"1"}]},{"pgid":"4.c","version":"165'145","reported_seq":510,"reported_epoch":168,"state":"remapped+peering","last_fresh":"2024-04-20T02:53:07.192969+0000","last_change":"2024-04-20T02:53:07.193212+0000","last_active":"2024-04-20T02:53:07.192969+0000","last_peered":"2024-04-20T02:53:06.414101+0000","last_clean":"2024-04-20T02:52:47.201790+0000","last_became_active":"2024-04-20T02:52:52.155457+0000","last_became_peered":"2024-04-20T02:52:52.155457+0000","last_unstale":"2024-04-20T02:53:07.192969+0000","last_undegraded":"2024-04-20T02:53:07.192969+0000","last_fullsized":"2024-04-20T02:53:07.192969+0000","mapping_epoch":168,"log_start":"157'141","ondisk_log_start":"157'141","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'95","last_scrub_stamp":"2024-04-20T02:52:09.803923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:09.803923+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":141,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:15.784812+0000","scrub_duration":0.0026115750000000001,"objects_trimmed":0,"snaptrim_duration":0.013286788000000001,"stat_sum":{"num_bytes":11915840,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":149,"num_read_kb":161187,"num_write":86,"num_write_kb":23588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,6],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"41","length":"1"},{"start":"43","length":"1"}]},{"pgid":"4.d","version":"167'271","reported_seq":804,"reported_epoch":167,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:53:06.435072+0000","last_change":"2024-04-20T02:53:06.431743+0000","last_active":"2024-04-20T02:53:06.435072+0000","last_peered":"2024-04-20T02:53:06.435072+0000","last_clean":"2024-04-20T02:53:06.435072+0000","last_became_active":"2024-04-20T02:52:46.144032+0000","last_became_peered":"2024-04-20T02:52:46.144032+0000","last_unstale":"2024-04-20T02:53:06.435072+0000","last_undegraded":"2024-04-20T02:53:06.435072+0000","last_fullsized":"2024-04-20T02:53:06.435072+0000","mapping_epoch":147,"log_start":"163'267","ondisk_log_start":"163'267","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'169","last_scrub_stamp":"2024-04-20T02:52:08.841360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:08.841360+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":267,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:23.317619+0000","scrub_duration":0.002314502,"objects_trimmed":0,"snaptrim_duration":0.14318851499999999,"stat_sum":{"num_bytes":12689450,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":418,"num_read_kb":513736,"num_write":158,"num_write_kb":42985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":27793283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"}]},{"pgid":"4.e","version":"164'114","reported_seq":488,"reported_epoch":168,"state":"remapped","last_fresh":"2024-04-20T02:53:07.196662+0000","last_change":"2024-04-20T02:53:07.621136+0000","last_active":"2024-04-20T02:53:07.196662+0000","last_peered":"2024-04-20T02:53:03.163319+0000","last_clean":"2024-04-20T02:52:41.345488+0000","last_became_active":"2024-04-20T02:52:51.154816+0000","last_became_peered":"2024-04-20T02:52:51.154816+0000","last_unstale":"2024-04-20T02:53:07.196662+0000","last_undegraded":"2024-04-20T02:53:07.196662+0000","last_fullsized":"2024-04-20T02:53:07.196662+0000","mapping_epoch":168,"log_start":"161'110","ondisk_log_start":"161'110","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'77","last_scrub_stamp":"2024-04-20T02:51:46.088382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:46.088382+0000","objects_scrubbed":3,"log_size":4,"log_dups_size":110,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:14.376107+0000","scrub_duration":0.002170693,"objects_trimmed":0,"snaptrim_duration":0.0013274350000000001,"stat_sum":{"num_bytes":10202891,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":162,"num_read_kb":221957,"num_write":58,"num_write_kb":18271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"1"},{"start":"3a","length":"1"},{"start":"43","length":"1"}]},{"pgid":"4.f","version":"163'44","reported_seq":436,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-04-20T02:53:06.286040+0000","last_change":"2024-04-20T02:53:06.286040+0000","last_active":"2024-04-20T02:53:06.286040+0000","last_peered":"2024-04-20T02:53:06.286040+0000","last_clean":"2024-04-20T02:53:06.286040+0000","last_became_active":"2024-04-20T02:52:54.156070+0000","last_became_peered":"2024-04-20T02:52:54.156070+0000","last_unstale":"2024-04-20T02:53:06.286040+0000","last_undegraded":"2024-04-20T02:53:06.286040+0000","last_fullsized":"2024-04-20T02:53:06.286040+0000","mapping_epoch":154,"log_start":"153'40","ondisk_log_start":"153'40","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'36","last_scrub_stamp":"2024-04-20T02:52:46.138400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:46.138400+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":40,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:09.709596+0000","scrub_duration":0.003388247,"objects_trimmed":0,"snaptrim_duration":0.00049776300000000002,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":115234,"num_write":27,"num_write_kb":7507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1714357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"4.10","version":"163'136","reported_seq":735,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-04-20T02:53:06.415080+0000","last_change":"2024-04-20T02:53:06.415080+0000","last_active":"2024-04-20T02:53:06.415080+0000","last_peered":"2024-04-20T02:53:06.415080+0000","last_clean":"2024-04-20T02:53:06.415080+0000","last_became_active":"2024-04-20T02:52:46.140881+0000","last_became_peered":"2024-04-20T02:52:46.140881+0000","last_unstale":"2024-04-20T02:53:06.415080+0000","last_undegraded":"2024-04-20T02:53:06.415080+0000","last_fullsized":"2024-04-20T02:53:06.415080+0000","mapping_epoch":147,"log_start":"151'132","ondisk_log_start":"151'132","created":60,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'132","last_scrub_stamp":"2024-04-20T02:52:55.175686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:55.175686+0000","objects_scrubbed":3,"log_size":4,"log_dups_size":130,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.746787+0000","scrub_duration":0.0022427829999999999,"objects_trimmed":0,"snaptrim_duration":0.0013327160000000001,"stat_sum":{"num_bytes":6403533,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":202667,"num_write":25,"num_write_kb":7645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":17455143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"4.11","version":"154'212","reported_seq":823,"reported_epoch":169,"state":"remapped+peering","last_fresh":"2024-04-20T02:53:08.200223+0000","last_change":"2024-04-20T02:53:08.200421+0000","last_active":"2024-04-20T02:53:07.192890+0000","last_peered":"2024-04-20T02:53:06.181148+0000","last_clean":"2024-04-20T02:52:47.148809+0000","last_became_active":"2024-04-20T02:52:52.149357+0000","last_became_peered":"2024-04-20T02:52:52.149357+0000","last_unstale":"2024-04-20T02:53:08.200223+0000","last_undegraded":"2024-04-20T02:53:08.200223+0000","last_fullsized":"2024-04-20T02:53:08.200223+0000","mapping_epoch":169,"log_start":"153'209","ondisk_log_start":"153'209","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'147","last_scrub_stamp":"2024-04-20T02:51:44.085213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:44.085213+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":209,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022866549999999999,"objects_trimmed":0,"snaptrim_duration":0.0046705449999999999,"stat_sum":{"num_bytes":8161054,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":129,"num_read_kb":156949,"num_write":35,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":14392198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"41","length":"1"},{"start":"43","length":"1"}]},{"pgid":"4.12","version":"0'0","reported_seq":338,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-04-20T02:53:07.368147+0000","last_change":"2024-04-20T02:53:07.368147+0000","last_active":"2024-04-20T02:53:07.368147+0000","last_peered":"2024-04-20T02:53:07.368147+0000","last_clean":"2024-04-20T02:53:07.368147+0000","last_became_active":"2024-04-20T02:52:51.151032+0000","last_became_peered":"2024-04-20T02:52:51.151032+0000","last_unstale":"2024-04-20T02:53:07.368147+0000","last_undegraded":"2024-04-20T02:53:07.368147+0000","last_fullsized":"2024-04-20T02:53:07.368147+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:07.368016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:07.368016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:30.123817+0000","scrub_duration":0.003090041,"objects_trimmed":0,"snaptrim_duration":0.001738701,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"4.13","version":"170'283","reported_seq":940,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-04-20T02:53:09.349022+0000","last_change":"2024-04-20T02:53:06.183490+0000","last_active":"2024-04-20T02:53:09.349022+0000","last_peered":"2024-04-20T02:53:09.349022+0000","last_clean":"2024-04-20T02:53:09.349022+0000","last_became_active":"2024-04-20T02:52:52.145254+0000","last_became_peered":"2024-04-20T02:52:52.145254+0000","last_unstale":"2024-04-20T02:53:09.349022+0000","last_undegraded":"2024-04-20T02:53:09.349022+0000","last_fullsized":"2024-04-20T02:53:09.349022+0000","mapping_epoch":152,"log_start":"167'279","ondisk_log_start":"167'279","created":60,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"163'277","last_scrub_stamp":"2024-04-20T02:53:04.656168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.656168+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":279,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:21.498483+0000","scrub_duration":0.0029786719999999999,"objects_trimmed":0,"snaptrim_duration":0.0045733859999999996,"stat_sum":{"num_bytes":11860319,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":235,"num_read_kb":305407,"num_write":40,"num_write_kb":10935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":19243742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2f","length":"1"},{"start":"33","length":"1"},{"start":"35","length":"2"},{"start":"38","length":"3"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"43","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"}]},{"pgid":"4.14","version":"137'42","reported_seq":328,"reported_epoch":167,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:53:06.287712+0000","last_change":"2024-04-20T02:52:38.156023+0000","last_active":"2024-04-20T02:53:06.287712+0000","last_peered":"2024-04-20T02:53:06.287712+0000","last_clean":"2024-04-20T02:52:20.246719+0000","last_became_active":"2024-04-20T02:52:38.152272+0000","last_became_peered":"2024-04-20T02:52:38.152272+0000","last_unstale":"2024-04-20T02:53:06.287712+0000","last_undegraded":"2024-04-20T02:53:06.287712+0000","last_fullsized":"2024-04-20T02:53:06.287712+0000","mapping_epoch":139,"log_start":"117'39","ondisk_log_start":"117'39","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:50:34.220369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":39,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0034837169999999,"stat_sum":{"num_bytes":487453,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":23,"num_read_kb":26722,"num_write":15,"num_write_kb":3607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3776971,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":21,"stat_sum":{"num_bytes":195006383,"num_objects":87,"num_object_clones":44,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":16,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":4,"num_read":2600,"num_read_kb":3213689,"num_write":961,"num_write_kb":267383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":104888479,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":35,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":416571392,"data_stored":485281102,"data_compressed":856896,"data_compressed_allocated":663552,"data_compressed_original":2203648,"omap_allocated":2595,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":63,"acting":61,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":105424683,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":797,"num_read_kb":34136,"num_write":1095,"num_write_kb":306906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":53320659,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":222253056,"data_stored":237246721,"data_compressed":11644500,"data_compressed_allocated":12632064,"data_compressed_original":28119040,"omap_allocated":0,"internal_metadata":0},"log_size":20,"ondisk_log_size":20,"up":12,"acting":11,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509480,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92076,"kb_used_data":56852,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94279764,"statfs":{"total":96636764160,"available":96542478336,"internally_reserved":0,"allocated":58216448,"data_stored":63124328,"data_compressed":105745,"data_compressed_allocated":139264,"data_compressed_original":761856,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.442,"15min":0.442},"min":{"1min":0.27000000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57699999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.45900000000000002},"min":{"1min":0.32400000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.439}]},{"osd":1,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.53200000000000003,"15min":0.47799999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.9369999999999994,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.68500000000000005,"15min":0.64800000000000002},"min":{"1min":0.32400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":9.8119999999999994,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.40799999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.372},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.46300000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.61399999999999999,"15min":0.56200000000000006},"min":{"1min":0.35499999999999998,"5min":0.307,"15min":0.307},"max":{"1min":9.6440000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.56399999999999995,"15min":0.55000000000000004},"min":{"1min":0.29999999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.8599999999999999,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.48999999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.435,"15min":0.443},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65500000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.59399999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.497}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.441,"15min":0.45000000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57699999999999996,"5min":2.431,"15min":2.431},"last":0.34100000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.51500000000000001,"15min":0.52200000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.64600000000000002,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73299999999999998,"5min":1.599,"15min":1.599},"last":0.47899999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94296,"kb_used_data":62476,"kb_used_omap":1,"kb_used_meta":31806,"kb_avail":94277544,"statfs":{"total":96636764160,"available":96540205056,"internally_reserved":0,"allocated":63975424,"data_stored":67136005,"data_compressed":1886149,"data_compressed_allocated":2052096,"data_compressed_original":4476928,"omap_allocated":1657,"internal_metadata":32569735},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56599999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42299999999999999,"15min":0.41699999999999998},"min":{"1min":0.34699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.499}]},{"osd":2,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.45100000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.47899999999999998,"15min":0.46100000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.437,"15min":0.434},"min":{"1min":0.34499999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.36399999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.441}]},{"osd":6,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.27900000000000003,"5min":0.252,"15min":0.252},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.38400000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.441,"15min":0.437},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.39700000000000002}]}]},{"osd":2,"up_from":0,"seq":592705486855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141600,"kb_used_data":106388,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94230240,"statfs":{"total":96636764160,"available":96491765760,"internally_reserved":0,"allocated":108941312,"data_stored":124531323,"data_compressed":1131818,"data_compressed_allocated":1114112,"data_compressed_original":2707456,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.30199999999999999,"5min":0.34200000000000003,"15min":0.34799999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.42399999999999999,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.63600000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.26500000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.58999999999999997,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39700000000000002,"15min":0.39900000000000002},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.94299999999999995,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.753,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.36099999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.63600000000000001,"5min":1.097,"15min":1.097},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.53500000000000003,"15min":0.54900000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.56399999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.503}]},{"osd":6,"last update":"Sat Apr 20 02:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35999999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73599999999999999,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60099999999999998,"15min":0.58999999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":3.2719999999999998,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.72699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77900000000000003,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.60399999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":122080,"kb_used_data":86864,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94249760,"statfs":{"total":96636764160,"available":96511754240,"internally_reserved":0,"allocated":88948736,"data_stored":104825714,"data_compressed":1879302,"data_compressed_allocated":2043904,"data_compressed_original":4542464,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.29799999999999999,"15min":0.29899999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52500000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.377,"15min":0.38100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57999999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.47499999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.33300000000000002,"15min":0.33800000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.48399999999999999,"5min":0.751,"15min":0.751},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.373,"15min":0.379},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.496,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.254}]},{"osd":2,"last update":"Sat Apr 20 02:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":1.198,"15min":1.198},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.33200000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.85899999999999999,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.52900000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.01,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.46000000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47199999999999998,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69899999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.52700000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.54300000000000004,"15min":0.55500000000000005},"min":{"1min":0.34699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64100000000000001,"5min":1.153,"15min":1.153},"last":1.083},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.502}]},{"osd":7,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56399999999999995,"15min":0.57199999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90700000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.54800000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":133116,"kb_used_data":97896,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94238724,"statfs":{"total":96636764160,"available":96500453376,"internally_reserved":0,"allocated":100245504,"data_stored":112452321,"data_compressed":2741977,"data_compressed_allocated":3014656,"data_compressed_original":7290880,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.78700000000000003,"15min":0.84699999999999998},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.46000000000000002,"15min":0.439},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.496}]},{"osd":1,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48699999999999999,"15min":0.48399999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.0680000000000001,"5min":1.419,"15min":1.419},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58899999999999997,"15min":0.59899999999999998},"min":{"1min":0.32600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.756,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.36199999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.46600000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.69499999999999995,"15min":0.73499999999999999},"min":{"1min":0.372,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.59699999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.80800000000000005,"15min":0.85699999999999998},"min":{"1min":0.35499999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.1279999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.45100000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.625,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.57799999999999996,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.66100000000000003,"5min":1.038,"15min":1.038},"last":0.42499999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.47999999999999998,"15min":0.46899999999999997},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.59799999999999998,"15min":0.624},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68000000000000005,"5min":3.98,"15min":3.98},"last":0.441}]}]},{"osd":0,"up_from":15,"seq":64424509476,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159068,"kb_used_data":123852,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94212772,"statfs":{"total":96636764160,"available":96473878528,"internally_reserved":0,"allocated":126824448,"data_stored":138813054,"data_compressed":2905382,"data_compressed_allocated":3067904,"data_compressed_original":7434240,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36399999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.377,"15min":0.38200000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.81799999999999995,"5min":1.427,"15min":1.427},"last":0.32900000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.379},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.35499999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.42799999999999999,"15min":0.41199999999999998},"min":{"1min":0.23899999999999999,"5min":0.215,"15min":0.215},"max":{"1min":3.2730000000000001,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":0.47499999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90500000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.56200000000000006}]},{"osd":5,"last update":"Sat Apr 20 02:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69599999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.5349999999999999,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.44600000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.878,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.622},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.68799999999999994,"15min":0.70899999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3380000000000001,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.751}]}]},{"osd":4,"up_from":0,"seq":717259538434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54923264,"data_stored":58318136,"data_compressed":2657842,"data_compressed_allocated":2879488,"data_compressed_original":6397952,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18939904,"data_stored":19980867,"data_compressed":856197,"data_compressed_allocated":921600,"data_compressed_original":1998848,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12894208,"data_stored":13909415,"data_compressed":734140,"data_compressed_allocated":790528,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32878592,"data_stored":35196502,"data_compressed":1864770,"data_compressed_allocated":2027520,"data_compressed_original":4411392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18509824,"data_stored":19818665,"data_compressed":1037033,"data_compressed_allocated":1118208,"data_compressed_original":2482176,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32878592,"data_stored":35196502,"data_compressed":1864770,"data_compressed_allocated":2027520,"data_compressed_original":4411392,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":51228672,"data_stored":54826634,"data_compressed":2629748,"data_compressed_allocated":2867200,"data_compressed_original":6582272,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":80637952,"data_stored":90252461,"data_compressed":242222,"data_compressed_allocated":155648,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88211456,"data_stored":103727799,"data_compressed":275621,"data_compressed_allocated":192512,"data_compressed_original":708608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1507328,"data_stored":1501941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2595,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54280192,"data_stored":68806555,"data_compressed":14532,"data_compressed_allocated":16384,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56590336,"data_stored":67896718,"data_compressed":95804,"data_compressed_allocated":53248,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29282304,"data_stored":31104334,"data_compressed":21379,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":47173632,"data_stored":56274102,"data_compressed":106911,"data_compressed_allocated":114688,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":58888192,"data_stored":65717192,"data_compressed":100427,"data_compressed_allocated":106496,"data_compressed_original":249856,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:53:11.094 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-04-20T02:53:11.117 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 144.57256123299999, 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.118 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:53:11.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:53:11.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 136.74401234499999, 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.124 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.125 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:53:11.126 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:53:11.127 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.128 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.129 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 136.727760163, 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:53:11.130 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.131 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.132 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.133 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:53:11.134 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 78.768061144000001, 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:11.135 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:53:11.136 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.137 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:53:11.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3872 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 779924~785847,stat] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.765657905000001, 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070878165000000007, 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3872 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350641+0000", 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351179+0000", 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053797199999999995 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351196+0000", 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.66e-05 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351209+0000", 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2952000000000001e-05 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351306+0000", 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7272e-05 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351322+0000", 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6416e-05 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358164+0000", 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068416019999999996 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.144 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358336+0000", 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017156499999999999 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358594+0000", 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025794499999999998 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389148+0000", 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030554773 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409688+0000", 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020539497 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420893+0000", 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011205503 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420934+0000", 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1171000000000003e-05 2024-04-20T02:53:11.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420998+0000", 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3386999999999995e-05 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421521+0000", 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052313200000000005 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.147 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3873 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 2078257~580112] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.765074619, 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075737411000000004, 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3873 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351225+0000", 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351614+0000", 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038825099999999999 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351625+0000", 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1173e-05 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351634+0000", 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0489999999999993e-06 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351696+0000", 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1885999999999997e-05 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351708+0000", 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2702e-05 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389193+0000", 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037484574999999999 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389303+0000", 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110188 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389563+0000", 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026034499999999999 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420846+0000", 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031283014999999997 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421592+0000", 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074529299999999995 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421632+0000", 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0677999999999999e-05 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426683+0000", 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050507119999999997 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426752+0000", 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8484000000000004e-05 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426964+0000", 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021193500000000001 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3874 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 3301257~278667] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:11.154 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.764639618, 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075720365999999997, 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3874 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351660+0000", 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351852+0000", 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019116900000000001 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351861+0000", 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6050000000000004e-06 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351869+0000", 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4780000000000002e-06 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351916+0000", 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7222999999999998e-05 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351927+0000", 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0791000000000001e-05 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409721+0000", 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057793887000000002 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409840+0000", 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119837 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.410063+0000", 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022257200000000001 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421678+0000", 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011615448 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421721+0000", 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2191000000000003e-05 2024-04-20T02:53:11.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.423989+0000", 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022682879999999998 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427051+0000", 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003061785 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427157+0000", 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106684 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427381+0000", 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022416299999999999 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3875 4.10 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (57) in=7b,truncate 3579924] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.764418457000001, 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075851731000000006, 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.161 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3875 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351886+0000", 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.561e-06 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351893+0000", 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6100000000000002e-06 2024-04-20T02:53:11.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351899+0000", 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8660000000000002e-06 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352002+0000", 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103811 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352012+0000", 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4110000000000002e-06 2024-04-20T02:53:11.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421749+0000", 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.069736690000000004 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421861+0000", 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112587 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422120+0000", 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025878099999999997 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424547+0000", 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024270379999999999 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424960+0000", 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041259899999999998 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426257+0000", 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001297231 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427489+0000", 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012320949999999999 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427562+0000", 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2825999999999997e-05 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427734+0000", 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017230699999999999 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3876 4.10 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 26.764396551000001, 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076230629999999994, 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3876 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351906+0000", 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6350000000000002e-06 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351912+0000", 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8209999999999998e-06 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351918+0000", 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3839999999999996e-06 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352035+0000", 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117207 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352043+0000", 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0169999999999996e-06 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422511+0000", 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070467815000000003 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422619+0000", 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010819600000000001 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427772+0000", 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005153398 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427900+0000", 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012740399999999999 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.428135+0000", 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235138 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4067 4.c 4:392c1e2f:::smithi10548823-44:head [write 783292~540882] snapc 47=[47,45,3f] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.973704342000001, 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072910900000000001, 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.173 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4067 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142595+0000", 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142941+0000", 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034585799999999999 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142954+0000", 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3491e-05 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142965+0000", 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1139e-05 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143003+0000", 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7499000000000001e-05 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143011+0000", 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5490000000000006e-06 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156033+0000", 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013021576999999999 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156242+0000", 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020885500000000001 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156474+0000", 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195900000000001 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177835+0000", 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021361594000000001 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178072+0000", 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000237304 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.214963+0000", 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036890158999999999 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215046+0000", 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3399000000000006e-05 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215507+0000", 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046127700000000001 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4068 4.c 4:392c1e2f:::smithi10548823-44:head [write 1932766~507787] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.973325399, 2024-04-20T02:53:11.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073156952999999997, 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4068 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143300+0000", 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000325225 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143308+0000", 2024-04-20T02:53:11.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3860000000000007e-06 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143315+0000", 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7000000000000002e-06 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143360+0000", 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5475000000000001e-05 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143373+0000", 2024-04-20T02:53:11.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2204e-05 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178109+0000", 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034736547999999999 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178222+0000", 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112959 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178452+0000", 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000229686 2024-04-20T02:53:11.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215864+0000", 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037412038000000002 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215878+0000", 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4029000000000001e-05 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215929+0000", 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0738000000000001e-05 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215989+0000", 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0350999999999999e-05 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216132+0000", 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014332099999999999 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4069 4.c 4:392c1e2f:::smithi10548823-44:head [write 3210350~572013] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:11.185 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.972976586000001, 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108473219, 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4069 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143664+0000", 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034090500000000002 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143673+0000", 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1119999999999996e-06 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143679+0000", 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3589999999999996e-06 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143724+0000", 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4974e-05 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143734+0000", 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0275000000000001e-05 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197473+0000", 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053738971000000003 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197616+0000", 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014275900000000001 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197834+0000", 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021806000000000001 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251517+0000", 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053683445000000003 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251579+0000", 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1255999999999998e-05 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251590+0000", 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1399e-05 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251657+0000", 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7039999999999995e-05 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251797+0000", 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000140235 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4070 4.c 4:392c1e2f:::smithi10548823-44:head [setxattr _header (57) in=7b,truncate 3983292] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.972613798000001, 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108344988, 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.192 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4070 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143686+0000", 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143690+0000", 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7579999999999998e-06 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143697+0000", 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0119999999999999e-06 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143702+0000", 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.78e-06 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143758+0000", 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640999999999998e-05 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143766+0000", 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9519999999999994e-06 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216172+0000", 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072406561999999994 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216295+0000", 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122452 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216531+0000", 2024-04-20T02:53:11.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023635999999999999 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251815+0000", 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035283760999999997 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251856+0000", 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1270999999999998e-05 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251892+0000", 2024-04-20T02:53:11.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251929+0000", 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7055999999999999e-05 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252032+0000", 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103183 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4071 4.c 4:392c1e2f:::smithi10548823-44:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.972578598999998, 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108670215, 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4071 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143721+0000", 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143727+0000", 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2510000000000002e-06 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143739+0000", 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2649e-05 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143749+0000", 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3340000000000008e-06 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143788+0000", 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9773e-05 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143796+0000", 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8720000000000002e-06 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216846+0000", 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073049823 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216948+0000", 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101525 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252070+0000", 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035122911999999999 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252180+0000", 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010907499999999999 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252392+0000", 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021285400000000001 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4106 4.c 4:3d524298:::smithi10548823-18:head [write 484134~708729] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.972545493999998, 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108955254, 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.204 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4106 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:11.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144188+0000", 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000433771 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144198+0000", 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4970000000000007e-06 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144205+0000", 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3409999999999997e-06 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145714+0000", 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015088110000000001 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145727+0000", 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2989e-05 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217011+0000", 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071283814000000001 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217168+0000", 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015707300000000001 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217397+0000", 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022881900000000001 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251870+0000", 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034473539999999997 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252459+0000", 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000589151 2024-04-20T02:53:11.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252511+0000", 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1282999999999997e-05 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252569+0000", 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8294999999999997e-05 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252711+0000", 2024-04-20T02:53:11.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000141435 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4107 4.c 4:3d524298:::smithi10548823-18:head [write 1883658~200476] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.972083429000001, 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.10880477099999999, 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.211 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4107 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144216+0000", 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144346+0000", 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000129732 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144355+0000", 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4300000000000006e-06 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144361+0000", 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9150000000000001e-06 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145754+0000", 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0013929649999999999 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145763+0000", 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2990000000000002e-06 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243874+0000", 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.098111462999999996 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243990+0000", 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0001156 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.244226+0000", 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235839 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252760+0000", 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085345439999999998 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252817+0000", 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6842000000000002e-05 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252829+0000", 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1817999999999999e-05 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252884+0000", 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4610000000000001e-05 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253022+0000", 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013853800000000001 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4108 4.c 4:3d524298:::smithi10548823-18:head [setxattr _header (57) in=7b,truncate 2084134] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:11.217 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.971935004999999, 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.120324168, 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4108 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144365+0000", 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144369+0000", 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8500000000000004e-06 2024-04-20T02:53:11.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144374+0000", 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0080000000000002e-06 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144379+0000", 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8239999999999999e-06 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145790+0000", 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014115449999999999 2024-04-20T02:53:11.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145800+0000", 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0429999999999996e-06 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253101+0000", 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.107301173 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253210+0000", 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109146 2024-04-20T02:53:11.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253456+0000", 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024599700000000002 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254623+0000", 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001166718 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.258220+0000", 2024-04-20T02:53:11.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003597135 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264486+0000", 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062665819999999997 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264558+0000", 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1364000000000003e-05 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264690+0000", 2024-04-20T02:53:11.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013227099999999999 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4109 4.c 4:3d524298:::smithi10548823-18:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.971917482999999, 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12051582399999999, 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.224 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4109 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144385+0000", 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2869999999999999e-06 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144392+0000", 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5749999999999997e-06 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144397+0000", 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3589999999999999e-06 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145829+0000", 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00143191 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145838+0000", 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:53:11.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254052+0000", 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108214029 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254156+0000", 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010343799999999999 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264726+0000", 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010570144 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264796+0000", 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9886000000000001e-05 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264899+0000", 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103336 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:4146 3.1 e153/152 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head v 153'228, mlcod=0'0)", 2024-04-20T02:53:11.229 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.934521297, 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066047175, 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:11.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181778+0000", 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181997+0000", 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021896 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182011+0000", 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3339999999999999e-05 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182021+0000", 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.046e-05 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182068+0000", 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6931999999999997e-05 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182104+0000", 2024-04-20T02:53:11.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6321e-05 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247786+0000", 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065681787000000005 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247827+0000", 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0491000000000002e-05 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4183 3.2 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head [write 0~2392064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 17.464887125000001, 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064859071000000004, 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4183 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:11.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651396+0000", 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999819 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652285+0000", 2024-04-20T02:53:11.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088949500000000002 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652306+0000", 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0812e-05 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652333+0000", 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7086000000000001e-05 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652373+0000", 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9690000000000001e-05 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652499+0000", 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012642299999999999 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652608+0000", 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108661 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.702325+0000", 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049717048999999999 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.711665+0000", 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093403370000000006 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716188+0000", 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004522997 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716227+0000", 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9240999999999999e-05 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:11.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:11.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:11.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716273+0000", 2024-04-20T02:53:11.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5228000000000001e-05 2024-04-20T02:53:11.239 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.239 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.239 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.239 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:11.239 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:11.239 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:11.241 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:1915: set_chunk oid 40 offset: 293376 length: 877056 target oid cc338803e3059bf33c1cf5e0ebaa587bab7749a167ec37a3185c75f2c9120a87 offset: 0 snap 39 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:1916: tier_promote oid 5 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:1905: finishing write tid 4 to smithi10548823-44 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:1905: oid 44 updating version 0 to 149 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:1905: oid 44 version 149 is already newer than 147 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:1905: oid 44 version 149 is already newer than 148 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 44 v 149 (ObjNum 390 snap 44 seq_num 390) dirty exists 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:1905: left oid 44 (ObjNum 390 snap 44 seq_num 390) 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:1906: finishing write tid 1 to smithi10548823-37 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:1906: finishing write tid 2 to smithi10548823-37 2024-04-20T02:53:11.242 INFO:tasks.rados.rados.0.smithi105.stdout:1906: finishing write tid 3 to smithi10548823-37 2024-04-20T02:53:11.243 INFO:tasks.rados.rados.0.smithi105.stdout:1906: finishing write tid 4 to smithi10548823-37 2024-04-20T02:53:11.243 INFO:tasks.rados.rados.0.smithi105.stdout:1906: oid 37 updating version 0 to 112 2024-04-20T02:53:11.243 INFO:tasks.rados.rados.0.smithi105.stdout:1906: oid 37 version 112 is already newer than 111 2024-04-20T02:53:11.243 INFO:tasks.rados.rados.0.smithi105.stdout:1906: oid 37 version 112 is already newer than 110 2024-04-20T02:53:11.243 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 112 (ObjNum 391 snap 44 seq_num 391) dirty exists 2024-04-20T02:53:11.243 INFO:tasks.rados.rados.0.smithi105.stdout:1906: left oid 37 (ObjNum 391 snap 44 seq_num 391) 2024-04-20T02:53:11.243 INFO:tasks.rados.rados.0.smithi105.stdout:1913: expect (ObjNum 383 snap 41 seq_num 383) 2024-04-20T02:53:11.243 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:11.241+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:11.342 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:11.342+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:11.373 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:11.374 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:11.374 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:11.374 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:11.384 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:1911: finishing write tid 1 to smithi10548823-4 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:1911: finishing write tid 2 to smithi10548823-4 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:1911: finishing write tid 3 to smithi10548823-4 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:1911: finishing write tid 4 to smithi10548823-4 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:1911: oid 4 updating version 0 to 283 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:1911: oid 4 version 283 is already newer than 281 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:1911: oid 4 version 283 is already newer than 282 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 283 (ObjNum 392 snap 44 seq_num 392) dirty exists 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:1911: left oid 4 (ObjNum 392 snap 44 seq_num 392) 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:1910: got (0) Success 2024-04-20T02:53:11.400 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 31 v 0 (ObjNum 366 snap 37 seq_num 366) dirty exists 2024-04-20T02:53:11.401 INFO:tasks.rados.rados.0.smithi105.stdout:1905: done (6 left) 2024-04-20T02:53:11.401 INFO:tasks.rados.rados.0.smithi105.stdout:1906: done (5 left) 2024-04-20T02:53:11.401 INFO:tasks.rados.rados.0.smithi105.stdout:1910: done (4 left) 2024-04-20T02:53:11.401 INFO:tasks.rados.rados.0.smithi105.stdout:1911: done (3 left) 2024-04-20T02:53:11.401 INFO:tasks.rados.rados.0.smithi105.stdout:1913: done (2 left) 2024-04-20T02:53:11.401 INFO:tasks.rados.rados.0.smithi105.stdout:1917: set_chunk oid 3 target oid 2024-04-20T02:53:11.428 INFO:tasks.rados.rados.0.smithi105.stdout:1917: set_chunk oid 3 offset: 268288 length: 1449984 target oid c724fbd835f8fe8da9383defb2b17293770fd7bf1cb2917021703ffb89fb2152 offset: 0 snap 39 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:1918: read oid 16 snap -1 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:1918: expect deleted 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:1919: tier_flush oid 2 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:1919: tier_flush oid 2 snap -1 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:1915: finishing set_chunk to oid 40 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 40 set_chunk returned error code -95 offset: 293376 length: 877056 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 40 v 0 (ObjNum 386 snap 41 seq_num 386) dirty exists 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:1915: done (4 left) 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:1920: read oid 33 snap -1 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:1920: expect (ObjNum 380 snap 41 seq_num 380) 2024-04-20T02:53:11.429 INFO:tasks.rados.rados.0.smithi105.stdout:1921: set_chunk oid 40 target oid 2024-04-20T02:53:11.443 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:11.443+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:11.460 INFO:tasks.rados.rados.0.smithi105.stdout:1921: set_chunk oid 40 offset: 1885184 length: 367104 target oid 4b18af7e8da28cd55b15bce10cb42094ac9710621b13431f3aaedc39c53ee71d offset: 0 snap 42 2024-04-20T02:53:11.460 INFO:tasks.rados.rados.0.smithi105.stdout:1922: rollback oid 25 current snap is 44 2024-04-20T02:53:11.460 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 25 to 42 2024-04-20T02:53:11.461 INFO:tasks.rados.rados.0.smithi105.stdout:1923: write (excl) oid 11 current snap is 44 2024-04-20T02:53:11.461 INFO:tasks.rados.rados.0.smithi105.stdout:1923: seq_num 394 ranges {621958=782109,2089670=532288} 2024-04-20T02:53:11.464 INFO:tasks.rados.rados.0.smithi105.stdout:1923: writing smithi10548823-11 from 621958 to 1404067 tid 1 2024-04-20T02:53:11.471 INFO:tasks.rados.rados.0.smithi105.stdout:1923: writing smithi10548823-11 from 2089670 to 2621958 tid 2 2024-04-20T02:53:11.472 INFO:tasks.rados.rados.0.smithi105.stdout:1924: snap_remove snap 43 2024-04-20T02:53:11.521 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:11.521 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:11.521 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:11.521 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:11.522 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:11.532 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:11.575 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:11.576+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:11.644 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:11.644+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:11.661 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.925631017, 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:11.662 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:53:11.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:53:11.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:53:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:53:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:53:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:11.668 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.89434195600001, 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:53:11.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:53:11.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:53:11.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.88965150000001, 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:53:11.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:53:11.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:53:11.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.680 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.882895031, 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:11.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:11.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:53:11.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:53:11.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:11.686 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.87307952, 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:53:11.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:53:11.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:53:11.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.87203558799999, 2024-04-20T02:53:11.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:53:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:53:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:11.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:11.698 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.872014428, 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:53:11.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:53:11.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.664443847, 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.703 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:11.707 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.650249202, 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:53:11.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.64428194499999, 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.715 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.63587594200001, 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:53:11.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:11.719 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.53224390099999, 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:53:11.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:11.724 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.30164759100001, 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:53:11.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:53:11.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.728 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.274703442, 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:11.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.732 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.249345175, 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:53:11.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:53:11.736 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.221204731, 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.18383612599999, 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:53:11.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:53:11.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:53:11.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.18353920300001, 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:53:11.745 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.747 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.17845608499999, 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:53:11.749 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 149.15145352600001, 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:53:11.753 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:11.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:11.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:11.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:11.757 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:11.759 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:11.759 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:11.759+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:11.859 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:11.859+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:11.928 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:11.928 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:11.928 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:11.928 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:11.938 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:11.959 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:11.959+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:12.060 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:12.060+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:12.106 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:12.107 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:12.107 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:12.107 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:12.107 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:12.117 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:12.160 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:12.161+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:12.211 INFO:tasks.rados.rados.0.smithi105.stdout:1924: done (8 left) 2024-04-20T02:53:12.211 INFO:tasks.rados.rados.0.smithi105.stdout:1925: set_chunk oid 43 target oid 2024-04-20T02:53:12.227 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:53:12.230 INFO:tasks.rados.rados.0.smithi105.stdout:1925: set_chunk oid 43 offset: 884736 length: 567296 target oid 91499e28d4f38f64ad8b2f0ce90ed7b234a9cc419db87793690bc70ccd223613 offset: 0 snap -1 2024-04-20T02:53:12.230 INFO:tasks.rados.rados.0.smithi105.stdout:1926: snap_remove snap 41 2024-04-20T02:53:12.252 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:12.252 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:12.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:12.252 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:12.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.252 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:53:12.252 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 172.64792292000001, 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:12.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:53:12.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 150.77075348, 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:53:12.259 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 150.410967828, 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:12.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:53:12.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:53:12.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 150.40060422299999, 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:53:12.269 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:53:12.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:53:12.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 132.110961519, 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.274 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:53:12.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.279 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 132.10582440900001, 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:53:12.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:53:12.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:53:12.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 132.105551397, 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:12.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:12.291 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 132.10552201199999, 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:53:12.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:53:12.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:53:12.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 109.41819471300001, 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:53:12.296 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.297 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:12.300 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 83.970714490999995, 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:53:12.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:53:12.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 83.953446056999994, 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 60.780300261999997, 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:53:12.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:53:12.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:53:12.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3881 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'232, mlcod=141'228)", 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.833177721999999, 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050172267999999999, 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418659+0000", 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419496+0000", 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00083715799999999996 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419513+0000", 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7489000000000001e-05 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419524+0000", 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0118e-05 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429881+0000", 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010357762 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429905+0000", 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3546000000000001e-05 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468790+0000", 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038884821 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468832+0000", 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2341e-05 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3898 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'133, mlcod=133'129)", 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.781018750000001, 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038834664999999997, 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470817+0000", 2024-04-20T02:53:12.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471051+0000", 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023384600000000001 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471066+0000", 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4205e-05 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471074+0000", 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4090000000000003e-06 2024-04-20T02:53:12.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472816+0000", 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017419429999999999 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472843+0000", 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7296000000000001e-05 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509592+0000", 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036748496999999998 2024-04-20T02:53:12.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509653+0000", 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1707999999999999e-05 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3899 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'134, mlcod=133'129)", 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.763752314000001, 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038218993999999999, 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.321 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.322 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488084+0000", 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488262+0000", 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017768600000000001 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488274+0000", 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2388000000000001e-05 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488283+0000", 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3840000000000003e-06 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489094+0000", 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081045100000000001 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489119+0000", 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4751000000000001e-05 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526264+0000", 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037144912000000002 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526304+0000", 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0513000000000003e-05 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.325 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3891 4.7 e146/145 4:e2c0d65b:::smithi10548823-46:head v 146'105, mlcod=143'100)", 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.762772218999999, 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041132308999999999, 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:12.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489072+0000", 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9539999999999998e-06 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489082+0000", 2024-04-20T02:53:12.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0329000000000001e-05 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489089+0000", 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4959999999999999e-06 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489814+0000", 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072423699999999997 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489837+0000", 2024-04-20T02:53:12.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3218999999999999e-05 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530157+0000", 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040319762000000002 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530198+0000", 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0886000000000001e-05 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.7 146/145 [PushOp(4:e2c0d65b:::smithi10548823-46:43, version: 143'96, data_included: [426944~820638,1468778~227938], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:e2c0d65b:::smithi10548823-46:43@143'96, size: 3311012, copy_subset: [0~3311012], clone_subset: {}, snapset: 45=[]:{3f=[3f,39],43=[41],45=[45]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1696716, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.761560867, 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039876652999999998, 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490276+0000", 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490785+0000", 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050941300000000001 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490804+0000", 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9080999999999999e-05 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490823+0000", 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.855e-05 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498138+0000", 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007315004 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498153+0000", 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5515e-05 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530153+0000", 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031999845999999998 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4016 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'146, mlcod=147'143)", 2024-04-20T02:53:12.333 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.989256620999999, 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054600045999999999, 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262580+0000", 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262733+0000", 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015289700000000001 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262746+0000", 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.375e-05 2024-04-20T02:53:12.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262753+0000", 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6159999999999998e-06 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262817+0000", 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4202999999999995e-05 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262842+0000", 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5296e-05 2024-04-20T02:53:12.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317137+0000", 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054294664999999999 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317181+0000", 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.337 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4017 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'147, mlcod=147'143)", 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.974209935000001, 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039585437000000001, 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:12.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277626+0000", 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278058+0000", 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043228299999999999 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278074+0000", 2024-04-20T02:53:12.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5775000000000001e-05 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278083+0000", 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2620000000000004e-06 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286849+0000", 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087652210000000001 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286881+0000", 2024-04-20T02:53:12.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1919999999999999e-05 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317188+0000", 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030307392999999998 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317213+0000", 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4953999999999999e-05 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4340 4.8 e163/114 4:1a8c7fb2:::smithi10548823-47:head v 163'162, mlcod=159'158)", 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 9.3395335500000005, 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.04004311, 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:12.342 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912303+0000", 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912722+0000", 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00041911999999999999 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912738+0000", 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6294e-05 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912748+0000", 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6059999999999992e-06 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912840+0000", 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2143000000000002e-05 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912867+0000", 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7121e-05 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952298+0000", 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039430999000000001 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952347+0000", 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8990999999999999e-05 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.345 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.346 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.346 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:12.346 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:12.346 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:12.347 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:12.348 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:12.348+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:12.448 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:12.449+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:12.518 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:12.518 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:12.518 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:12.518 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:12.528 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:12.549 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:12.549+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:12.620 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:53:12.650 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:12.651+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:12.696 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:12.697 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:12.697 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:12.697 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:12.697 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:12.706 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:12.751 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:12.751+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:12.763 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:12.782 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:12.801 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:12.819 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:12.838 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:12.851 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:12.852+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:12.857 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:12.875 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:12.876 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:12.876 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:12.887 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:12.893 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:12.895 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:12.895 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:12.895 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:12.895 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:12.895 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:12.895 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:12.896 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:12.896 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:12.953 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:12.954+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:13.014 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:13.015 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:13.025 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:13.069 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:13.069+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:13.152 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:13.153 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:13.154 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:13.154+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:13.165 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:13.214 INFO:tasks.rados.rados.0.smithi105.stdout:1926: done (9 left) 2024-04-20T02:53:13.214 INFO:tasks.rados.rados.0.smithi105.stdout:1927: tier_flush oid 21 2024-04-20T02:53:13.214 INFO:tasks.rados.rados.0.smithi105.stdout:1927: tier_flush oid 21 snap -1 2024-04-20T02:53:13.215 INFO:tasks.rados.rados.0.smithi105.stdout:1916: got (0) Success 2024-04-20T02:53:13.215 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 5 v 212 (ObjNum 368 snap 37 seq_num 368) dirty exists 2024-04-20T02:53:13.215 INFO:tasks.rados.rados.0.smithi105.stdout:1920: expect (ObjNum 380 snap 41 seq_num 380) 2024-04-20T02:53:13.217 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:53:13.255 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:13.255+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:13.338 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:13.339 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:13.347 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:13.355 INFO:tasks.rados.rados.0.smithi105.stdout:1917: finishing set_chunk to oid 3 2024-04-20T02:53:13.355 INFO:tasks.rados.rados.0.smithi105.stdout:1922: finishing rollback tid 0 to smithi10548823-25 2024-04-20T02:53:13.356 INFO:tasks.rados.rados.0.smithi105.stdout:1922: finishing rollback tid 1 to smithi10548823-25 2024-04-20T02:53:13.356 INFO:tasks.rados.rados.0.smithi105.stdout:1922: finishing rollback tid 2 to smithi10548823-25 2024-04-20T02:53:13.356 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 25 v 315 (ObjNum 383 snap 41 seq_num 383) dirty exists 2024-04-20T02:53:13.356 INFO:tasks.rados.rados.0.smithi105.stdout:1919: got (0) Success 2024-04-20T02:53:13.356 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 307 snap 33 seq_num 307) dirty exists 2024-04-20T02:53:13.356 INFO:tasks.rados.rados.0.smithi105.stdout:1921: finishing set_chunk to oid 40 2024-04-20T02:53:13.357 INFO:tasks.rados.rados.0.smithi105.stdout:1923: finishing write tid 1 to smithi10548823-11 2024-04-20T02:53:13.357 INFO:tasks.rados.rados.0.smithi105.stdout:1923: finishing write tid 2 to smithi10548823-11 2024-04-20T02:53:13.357 INFO:tasks.rados.rados.0.smithi105.stdout:1923: finishing write tid 3 to smithi10548823-11 2024-04-20T02:53:13.357 INFO:tasks.rados.rados.0.smithi105.stdout:1923: finishing write tid 4 to smithi10548823-11 2024-04-20T02:53:13.357 INFO:tasks.rados.rados.0.smithi105.stdout:1923: oid 11 updating version 0 to 181 2024-04-20T02:53:13.357 INFO:tasks.rados.rados.0.smithi105.stdout:1923: oid 11 version 181 is already newer than 180 2024-04-20T02:53:13.357 INFO:tasks.rados.rados.0.smithi105.stdout:1923: oid 11 version 181 is already newer than 179 2024-04-20T02:53:13.357 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 181 (ObjNum 394 snap 44 seq_num 394) dirty exists 2024-04-20T02:53:13.358 INFO:tasks.rados.rados.0.smithi105.stdout:1923: left oid 11 (ObjNum 394 snap 44 seq_num 394) 2024-04-20T02:53:13.358 INFO:tasks.rados.rados.0.smithi105.stdout:1925: finishing set_chunk to oid 43 2024-04-20T02:53:13.358 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 111 (ObjNum 276 snap 26 seq_num 276) dirty exists 2024-04-20T02:53:13.358 INFO:tasks.rados.rados.0.smithi105.stdout:1916: done (9 left) 2024-04-20T02:53:13.358 INFO:tasks.rados.rados.0.smithi105.stdout:1917: done (8 left) 2024-04-20T02:53:13.358 INFO:tasks.rados.rados.0.smithi105.stdout:1918: done (7 left) 2024-04-20T02:53:13.358 INFO:tasks.rados.rados.0.smithi105.stdout:1919: done (6 left) 2024-04-20T02:53:13.359 INFO:tasks.rados.rados.0.smithi105.stdout:1920: done (5 left) 2024-04-20T02:53:13.359 INFO:tasks.rados.rados.0.smithi105.stdout:1921: done (4 left) 2024-04-20T02:53:13.359 INFO:tasks.rados.rados.0.smithi105.stdout:1922: done (3 left) 2024-04-20T02:53:13.359 INFO:tasks.rados.rados.0.smithi105.stdout:1923: done (2 left) 2024-04-20T02:53:13.359 INFO:tasks.rados.rados.0.smithi105.stdout:1925: done (1 left) 2024-04-20T02:53:13.359 INFO:tasks.rados.rados.0.smithi105.stdout:1928: set_chunk oid 39 target oid 2024-04-20T02:53:13.392 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:13.393+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:13.395 INFO:tasks.rados.rados.0.smithi105.stdout:1928: set_chunk oid 39 offset: 1389568 length: 1278464 target oid baf17e4f3ea70e2dbedadc461000e85bce1dc214f15773e242064b4d45eadce3 offset: 0 snap -1 2024-04-20T02:53:13.395 INFO:tasks.rados.rados.0.smithi105.stdout:1929: tier_flush oid 41 2024-04-20T02:53:13.395 INFO:tasks.rados.rados.0.smithi105.stdout:1929: tier_flush oid 41 snap -1 2024-04-20T02:53:13.395 INFO:tasks.rados.rados.0.smithi105.stdout:1927: got (16) Device or resource busy 2024-04-20T02:53:13.396 INFO:tasks.rados.rados.0.smithi105.stdout:check_oldest_snap_flushed oid 21 v 177 (ObjNum 382 snap 41 seq_num 382) unflushed 42 2024-04-20T02:53:13.396 INFO:tasks.rados.rados.0.smithi105.stdout:1927: done (2 left) 2024-04-20T02:53:13.396 INFO:tasks.rados.rados.0.smithi105.stdout:1930: read oid 1 snap -1 2024-04-20T02:53:13.396 INFO:tasks.rados.rados.0.smithi105.stdout:1930: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:53:13.396 INFO:tasks.rados.rados.0.smithi105.stdout:1931: set_chunk oid 34 target oid 2024-04-20T02:53:13.404 INFO:tasks.rados.rados.0.smithi105.stdout:1931: set_chunk oid 34 offset: 260608 length: 324096 target oid 79d7b14e7f9951a954a1857757e609823a963d0963c5e0d76ed4898932118774 offset: 0 snap -1 2024-04-20T02:53:13.404 INFO:tasks.rados.rados.0.smithi105.stdout:1932: read oid 15 snap 42 2024-04-20T02:53:13.405 INFO:tasks.rados.rados.0.smithi105.stdout:1932: expect deleted 2024-04-20T02:53:13.405 INFO:tasks.rados.rados.0.smithi105.stdout:1933: read oid 4 snap 39 2024-04-20T02:53:13.405 INFO:tasks.rados.rados.0.smithi105.stdout:1933: expect (ObjNum 351 snap 37 seq_num 351) 2024-04-20T02:53:13.405 INFO:tasks.rados.rados.0.smithi105.stdout:1934: read oid 37 snap -1 2024-04-20T02:53:13.405 INFO:tasks.rados.rados.0.smithi105.stdout:1934: expect (ObjNum 391 snap 44 seq_num 391) 2024-04-20T02:53:13.405 INFO:tasks.rados.rados.0.smithi105.stdout:1935: read oid 50 snap -1 2024-04-20T02:53:13.405 INFO:tasks.rados.rados.0.smithi105.stdout:1935: expect (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:53:13.405 INFO:tasks.rados.rados.0.smithi105.stdout:1936: read oid 19 snap -1 2024-04-20T02:53:13.405 INFO:tasks.rados.rados.0.smithi105.stdout:1936: expect (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:53:13.408 INFO:tasks.rados.rados.0.smithi105.stdout:1937: write (excl) oid 6 current snap is 44 2024-04-20T02:53:13.408 INFO:tasks.rados.rados.0.smithi105.stdout:1937: seq_num 395 ranges {638765=457358,1748152=604803} 2024-04-20T02:53:13.410 INFO:tasks.rados.rados.0.smithi105.stdout:1937: writing smithi10548823-6 from 638765 to 1096123 tid 1 2024-04-20T02:53:13.431 INFO:tasks.rados.rados.0.smithi105.stdout:1937: writing smithi10548823-6 from 1748152 to 2352955 tid 2 2024-04-20T02:53:13.431 INFO:tasks.rados.rados.0.smithi105.stdout:1928: finishing set_chunk to oid 39 2024-04-20T02:53:13.431 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 39 set_chunk returned error code -95 offset: 1389568 length: 1278464 2024-04-20T02:53:13.431 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 39 v 0 (ObjNum 283 snap 27 seq_num 283) dirty exists 2024-04-20T02:53:13.431 INFO:tasks.rados.rados.0.smithi105.stdout:1930: expect (ObjNum 249 snap 22 seq_num 249) 2024-04-20T02:53:13.462 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:13.462+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:13.491 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:13.491 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:13.502 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:13.508 INFO:tasks.rados.rados.0.smithi105.stdout:1933: expect (ObjNum 351 snap 37 seq_num 351) 2024-04-20T02:53:13.563 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:13.563+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:13.663 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:13.664+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:13.674 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:13.674 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:13.684 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:13.746 INFO:tasks.rados.rados.0.smithi105.stdout:1931: finishing set_chunk to oid 34 2024-04-20T02:53:13.746 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 34 v 277 (ObjNum 377 snap 40 seq_num 377) dirty exists 2024-04-20T02:53:13.746 INFO:tasks.rados.rados.0.smithi105.stdout:1928: done (9 left) 2024-04-20T02:53:13.746 INFO:tasks.rados.rados.0.smithi105.stdout:1930: done (8 left) 2024-04-20T02:53:13.746 INFO:tasks.rados.rados.0.smithi105.stdout:1931: done (7 left) 2024-04-20T02:53:13.746 INFO:tasks.rados.rados.0.smithi105.stdout:1932: done (6 left) 2024-04-20T02:53:13.747 INFO:tasks.rados.rados.0.smithi105.stdout:1933: done (5 left) 2024-04-20T02:53:13.747 INFO:tasks.rados.rados.0.smithi105.stdout:1938: snap_remove snap 39 2024-04-20T02:53:13.764 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:13.764+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:13.812 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:13.812 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:13.823 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:13.868 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:13.868+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:13.952 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:13.953 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:13.963 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:14.004 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:14.004+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:14.067 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:14.068+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:14.131 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:14.131 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:14.142 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:14.168 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:14.169+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:14.219 INFO:tasks.rados.rados.0.smithi105.stdout:1938: done (5 left) 2024-04-20T02:53:14.219 INFO:tasks.rados.rados.0.smithi105.stdout:1939: read oid 7 snap -1 2024-04-20T02:53:14.219 INFO:tasks.rados.rados.0.smithi105.stdout:1939: expect (ObjNum 359 snap 37 seq_num 359) 2024-04-20T02:53:14.220 INFO:tasks.rados.rados.0.smithi105.stdout:1940: snap_remove snap 42 2024-04-20T02:53:14.269 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:14.269+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:14.277 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:14.277 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:14.288 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:14.370 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:14.371+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:14.462 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:14.462 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:14.471 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:14.471+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:14.473 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:14.571 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:14.572+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:14.648 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:14.648 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:14.659 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:14.672 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:14.672+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:14.772 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:14.772+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:14.796 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:14.796 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:14.807 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:14.874 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:14.875+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:14.907 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:14.978 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:14.975+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:14.978 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:14.978 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:14.990 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:15.058 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:15.076 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:15.077+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:15.078 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:15.096 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:15.116 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:15.136 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:15.156 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:15.167 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:15.167 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:15.167 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:15.167 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:15.168 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:15.177 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:15.177+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:15.182 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:15.183 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:15.199 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:15.201 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:15.201 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:15.201 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:15.202 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:15.202 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:15.202 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:15.202 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:15.221 INFO:tasks.rados.rados.0.smithi105.stdout:1940: done (6 left) 2024-04-20T02:53:15.221 INFO:tasks.rados.rados.0.smithi105.stdout:1941: read oid 33 snap -1 2024-04-20T02:53:15.221 INFO:tasks.rados.rados.0.smithi105.stdout:1941: expect (ObjNum 380 snap 41 seq_num 380) 2024-04-20T02:53:15.222 INFO:tasks.rados.rados.0.smithi105.stdout:1942: snap_create 2024-04-20T02:53:15.222 INFO:tasks.rados.rados.0.smithi105.stdout:1934: expect (ObjNum 391 snap 44 seq_num 391) 2024-04-20T02:53:15.230 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:53:15.278 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:15.279+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:15.358 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:15.358 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:15.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:15.358 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 148.81262559300001, 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.359 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:53:15.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:53:15.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:53:15.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:53:15.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:15.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:53:15.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:53:15.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:53:15.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:53:15.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:53:15.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:53:15.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:53:15.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:53:15.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:53:15.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:53:15.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:53:15.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:15.366 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.366 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 140.98407670500001, 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.367 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:53:15.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:53:15.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:53:15.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:53:15.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:53:15.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:53:15.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:53:15.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:53:15.374 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.374 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.374 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 140.96782452299999, 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:15.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:15.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:53:15.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:53:15.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:53:15.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:53:15.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:53:15.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 83.008125504000006, 2024-04-20T02:53:15.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:15.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:15.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:53:15.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:53:15.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3872 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 779924~785847,stat] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 31.005722264999999, 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070878165000000007, 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3872 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350641+0000", 2024-04-20T02:53:15.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351179+0000", 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053797199999999995 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351196+0000", 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.66e-05 2024-04-20T02:53:15.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351209+0000", 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2952000000000001e-05 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351306+0000", 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7272e-05 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351322+0000", 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6416e-05 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358164+0000", 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068416019999999996 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358336+0000", 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017156499999999999 2024-04-20T02:53:15.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358594+0000", 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025794499999999998 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389148+0000", 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030554773 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409688+0000", 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020539497 2024-04-20T02:53:15.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420893+0000", 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011205503 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420934+0000", 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1171000000000003e-05 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420998+0000", 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3386999999999995e-05 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421521+0000", 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052313200000000005 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3873 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 2078257~580112] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:15.395 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 31.005138979000002, 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075737411000000004, 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3873 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:15.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351225+0000", 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351614+0000", 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038825099999999999 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351625+0000", 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1173e-05 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351634+0000", 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0489999999999993e-06 2024-04-20T02:53:15.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351696+0000", 2024-04-20T02:53:15.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1885999999999997e-05 2024-04-20T02:53:15.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351708+0000", 2024-04-20T02:53:15.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2702e-05 2024-04-20T02:53:15.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389193+0000", 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037484574999999999 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389303+0000", 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110188 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389563+0000", 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026034499999999999 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420846+0000", 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031283014999999997 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421592+0000", 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074529299999999995 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421632+0000", 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0677999999999999e-05 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426683+0000", 2024-04-20T02:53:15.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050507119999999997 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426752+0000", 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8484000000000004e-05 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426964+0000", 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021193500000000001 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.403 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3874 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 3301257~278667] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 31.004703977999998, 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075720365999999997, 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.404 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3874 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351660+0000", 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351852+0000", 2024-04-20T02:53:15.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019116900000000001 2024-04-20T02:53:15.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351861+0000", 2024-04-20T02:53:15.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6050000000000004e-06 2024-04-20T02:53:15.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351869+0000", 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4780000000000002e-06 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351916+0000", 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7222999999999998e-05 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351927+0000", 2024-04-20T02:53:15.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0791000000000001e-05 2024-04-20T02:53:15.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409721+0000", 2024-04-20T02:53:15.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057793887000000002 2024-04-20T02:53:15.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409840+0000", 2024-04-20T02:53:15.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119837 2024-04-20T02:53:15.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:15.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.410063+0000", 2024-04-20T02:53:15.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022257200000000001 2024-04-20T02:53:15.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421678+0000", 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011615448 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421721+0000", 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2191000000000003e-05 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.423989+0000", 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022682879999999998 2024-04-20T02:53:15.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427051+0000", 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003061785 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427157+0000", 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106684 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427381+0000", 2024-04-20T02:53:15.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022416299999999999 2024-04-20T02:53:15.413 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.413 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.413 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.413 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3875 4.10 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (57) in=7b,truncate 3579924] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:15.413 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:15.414 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 31.004482817, 2024-04-20T02:53:15.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075851731000000006, 2024-04-20T02:53:15.414 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.414 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.414 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.414 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.414 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.414 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3875 2024-04-20T02:53:15.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.414 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351886+0000", 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.561e-06 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351893+0000", 2024-04-20T02:53:15.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6100000000000002e-06 2024-04-20T02:53:15.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351899+0000", 2024-04-20T02:53:15.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8660000000000002e-06 2024-04-20T02:53:15.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352002+0000", 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103811 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352012+0000", 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4110000000000002e-06 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421749+0000", 2024-04-20T02:53:15.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.069736690000000004 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421861+0000", 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112587 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422120+0000", 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025878099999999997 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424547+0000", 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024270379999999999 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424960+0000", 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041259899999999998 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426257+0000", 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001297231 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427489+0000", 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012320949999999999 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427562+0000", 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2825999999999997e-05 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427734+0000", 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017230699999999999 2024-04-20T02:53:15.421 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3876 4.10 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 31.004460910999999, 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076230629999999994, 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.422 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3876 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:15.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351906+0000", 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6350000000000002e-06 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351912+0000", 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8209999999999998e-06 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351918+0000", 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3839999999999996e-06 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352035+0000", 2024-04-20T02:53:15.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117207 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352043+0000", 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0169999999999996e-06 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422511+0000", 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070467815000000003 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422619+0000", 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010819600000000001 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427772+0000", 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005153398 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427900+0000", 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012740399999999999 2024-04-20T02:53:15.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.428135+0000", 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235138 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4067 4.c 4:392c1e2f:::smithi10548823-44:head [write 783292~540882] snapc 47=[47,45,3f] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.213768701999999, 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072910900000000001, 2024-04-20T02:53:15.428 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4067 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142595+0000", 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142941+0000", 2024-04-20T02:53:15.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034585799999999999 2024-04-20T02:53:15.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142954+0000", 2024-04-20T02:53:15.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3491e-05 2024-04-20T02:53:15.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142965+0000", 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1139e-05 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143003+0000", 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7499000000000001e-05 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143011+0000", 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5490000000000006e-06 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156033+0000", 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013021576999999999 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156242+0000", 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020885500000000001 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156474+0000", 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195900000000001 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177835+0000", 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021361594000000001 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178072+0000", 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000237304 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.214963+0000", 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036890158999999999 2024-04-20T02:53:15.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215046+0000", 2024-04-20T02:53:15.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3399000000000006e-05 2024-04-20T02:53:15.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215507+0000", 2024-04-20T02:53:15.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046127700000000001 2024-04-20T02:53:15.436 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4068 4.c 4:392c1e2f:::smithi10548823-44:head [write 1932766~507787] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.213389759000002, 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073156952999999997, 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.437 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4068 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143300+0000", 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000325225 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143308+0000", 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3860000000000007e-06 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143315+0000", 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7000000000000002e-06 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143360+0000", 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5475000000000001e-05 2024-04-20T02:53:15.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143373+0000", 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2204e-05 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178109+0000", 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034736547999999999 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178222+0000", 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112959 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178452+0000", 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000229686 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215864+0000", 2024-04-20T02:53:15.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037412038000000002 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215878+0000", 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4029000000000001e-05 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215929+0000", 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0738000000000001e-05 2024-04-20T02:53:15.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215989+0000", 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0350999999999999e-05 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216132+0000", 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014332099999999999 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.444 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4069 4.c 4:392c1e2f:::smithi10548823-44:head [write 3210350~572013] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.213040946, 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108473219, 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.445 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4069 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143664+0000", 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034090500000000002 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143673+0000", 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1119999999999996e-06 2024-04-20T02:53:15.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143679+0000", 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3589999999999996e-06 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143724+0000", 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4974e-05 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143734+0000", 2024-04-20T02:53:15.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0275000000000001e-05 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197473+0000", 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053738971000000003 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197616+0000", 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014275900000000001 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:15.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197834+0000", 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021806000000000001 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251517+0000", 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053683445000000003 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251579+0000", 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1255999999999998e-05 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251590+0000", 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1399e-05 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251657+0000", 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7039999999999995e-05 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251797+0000", 2024-04-20T02:53:15.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000140235 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4070 4.c 4:392c1e2f:::smithi10548823-44:head [setxattr _header (57) in=7b,truncate 3983292] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.212678157999999, 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108344988, 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.452 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4070 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:15.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143686+0000", 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143690+0000", 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7579999999999998e-06 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143697+0000", 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0119999999999999e-06 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143702+0000", 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.78e-06 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143758+0000", 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640999999999998e-05 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143766+0000", 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9519999999999994e-06 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216172+0000", 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072406561999999994 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216295+0000", 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122452 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216531+0000", 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023635999999999999 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251815+0000", 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035283760999999997 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251856+0000", 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1270999999999998e-05 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251892+0000", 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251929+0000", 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7055999999999999e-05 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252032+0000", 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103183 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.459 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.460 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.460 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4071 4.c 4:392c1e2f:::smithi10548823-44:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:15.460 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:15.460 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.212642959, 2024-04-20T02:53:15.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108670215, 2024-04-20T02:53:15.460 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.460 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4071 2024-04-20T02:53:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:15.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:15.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143721+0000", 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143727+0000", 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2510000000000002e-06 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143739+0000", 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2649e-05 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143749+0000", 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3340000000000008e-06 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143788+0000", 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9773e-05 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143796+0000", 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8720000000000002e-06 2024-04-20T02:53:15.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216846+0000", 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073049823 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216948+0000", 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101525 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252070+0000", 2024-04-20T02:53:15.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035122911999999999 2024-04-20T02:53:15.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252180+0000", 2024-04-20T02:53:15.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010907499999999999 2024-04-20T02:53:15.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252392+0000", 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021285400000000001 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4106 4.c 4:3d524298:::smithi10548823-18:head [write 484134~708729] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.212609854, 2024-04-20T02:53:15.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108955254, 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4106 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144188+0000", 2024-04-20T02:53:15.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000433771 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144198+0000", 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4970000000000007e-06 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144205+0000", 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3409999999999997e-06 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145714+0000", 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015088110000000001 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145727+0000", 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2989e-05 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217011+0000", 2024-04-20T02:53:15.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071283814000000001 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217168+0000", 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015707300000000001 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217397+0000", 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022881900000000001 2024-04-20T02:53:15.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251870+0000", 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034473539999999997 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252459+0000", 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000589151 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252511+0000", 2024-04-20T02:53:15.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1282999999999997e-05 2024-04-20T02:53:15.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252569+0000", 2024-04-20T02:53:15.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8294999999999997e-05 2024-04-20T02:53:15.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252711+0000", 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000141435 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4107 4.c 4:3d524298:::smithi10548823-18:head [write 1883658~200476] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.212147788999999, 2024-04-20T02:53:15.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.10880477099999999, 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4107 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:15.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144216+0000", 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144346+0000", 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000129732 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144355+0000", 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4300000000000006e-06 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144361+0000", 2024-04-20T02:53:15.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9150000000000001e-06 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145754+0000", 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0013929649999999999 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145763+0000", 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2990000000000002e-06 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243874+0000", 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.098111462999999996 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243990+0000", 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0001156 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.244226+0000", 2024-04-20T02:53:15.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235839 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252760+0000", 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085345439999999998 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252817+0000", 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6842000000000002e-05 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252829+0000", 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1817999999999999e-05 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252884+0000", 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4610000000000001e-05 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253022+0000", 2024-04-20T02:53:15.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013853800000000001 2024-04-20T02:53:15.483 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.483 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.483 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.483 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4108 4.c 4:3d524298:::smithi10548823-18:head [setxattr _header (57) in=7b,truncate 2084134] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:15.483 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:15.483 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.211999365, 2024-04-20T02:53:15.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.120324168, 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4108 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:15.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:15.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144365+0000", 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144369+0000", 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8500000000000004e-06 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144374+0000", 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0080000000000002e-06 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144379+0000", 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8239999999999999e-06 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145790+0000", 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014115449999999999 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145800+0000", 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0429999999999996e-06 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253101+0000", 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.107301173 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253210+0000", 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109146 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253456+0000", 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024599700000000002 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254623+0000", 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001166718 2024-04-20T02:53:15.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.258220+0000", 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003597135 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264486+0000", 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062665819999999997 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264558+0000", 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1364000000000003e-05 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264690+0000", 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013227099999999999 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.491 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4109 4.c 4:3d524298:::smithi10548823-18:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.211981843, 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12051582399999999, 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4109 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.492 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144385+0000", 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2869999999999999e-06 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144392+0000", 2024-04-20T02:53:15.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5749999999999997e-06 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144397+0000", 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3589999999999999e-06 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145829+0000", 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00143191 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145838+0000", 2024-04-20T02:53:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:53:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254052+0000", 2024-04-20T02:53:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108214029 2024-04-20T02:53:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:15.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254156+0000", 2024-04-20T02:53:15.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010343799999999999 2024-04-20T02:53:15.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264726+0000", 2024-04-20T02:53:15.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010570144 2024-04-20T02:53:15.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264796+0000", 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9886000000000001e-05 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264899+0000", 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103336 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.498 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:4146 3.1 e153/152 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head v 153'228, mlcod=0'0)", 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 23.174585657000002, 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066047175, 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181778+0000", 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:15.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181997+0000", 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021896 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182011+0000", 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3339999999999999e-05 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.501 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182021+0000", 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.046e-05 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182068+0000", 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6931999999999997e-05 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182104+0000", 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6321e-05 2024-04-20T02:53:15.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247786+0000", 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065681787000000005 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247827+0000", 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0491000000000002e-05 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.503 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4183 3.2 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head [write 0~2392064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 21.704951484999999, 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064859071000000004, 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:15.504 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4183 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:15.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651396+0000", 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999819 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652285+0000", 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088949500000000002 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652306+0000", 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0812e-05 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652333+0000", 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7086000000000001e-05 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:15.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652373+0000", 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9690000000000001e-05 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652499+0000", 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012642299999999999 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652608+0000", 2024-04-20T02:53:15.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108661 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.702325+0000", 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049717048999999999 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.711665+0000", 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093403370000000006 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716188+0000", 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004522997 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716227+0000", 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9240999999999999e-05 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:15.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716273+0000", 2024-04-20T02:53:15.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5228000000000001e-05 2024-04-20T02:53:15.511 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.511 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.511 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.511 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:15.511 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:15.511 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:15.513 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:15.513 INFO:tasks.rados.rados.0.smithi105.stdout:1935: expect (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:53:15.514 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:15.512+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:15.613 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:15.613+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:15.642 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:15.642 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:15.642 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:15.652 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:15.714 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:15.714+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:15.773 INFO:tasks.rados.rados.0.smithi105.stdout:1936: expect (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:53:15.780 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:15.780 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:15.780 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:15.780 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:15.780 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:15.793 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:15.831 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:15.832+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:15.872 INFO:tasks.rados.rados.0.smithi105.stdout:1929: got (0) Success 2024-04-20T02:53:15.872 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 41 v 0 (ObjNum 333 snap 36 seq_num 333) dirty exists 2024-04-20T02:53:15.873 INFO:tasks.rados.rados.0.smithi105.stdout:1937: finishing write tid 1 to smithi10548823-6 2024-04-20T02:53:15.873 INFO:tasks.rados.rados.0.smithi105.stdout:1937: finishing write tid 2 to smithi10548823-6 2024-04-20T02:53:15.873 INFO:tasks.rados.rados.0.smithi105.stdout:1937: finishing write tid 3 to smithi10548823-6 2024-04-20T02:53:15.873 INFO:tasks.rados.rados.0.smithi105.stdout:1937: finishing write tid 4 to smithi10548823-6 2024-04-20T02:53:15.873 INFO:tasks.rados.rados.0.smithi105.stdout:1937: oid 6 updating version 0 to 325 2024-04-20T02:53:15.873 INFO:tasks.rados.rados.0.smithi105.stdout:1937: oid 6 version 325 is already newer than 324 2024-04-20T02:53:15.873 INFO:tasks.rados.rados.0.smithi105.stdout:1937: oid 6 updating version 325 to 326 2024-04-20T02:53:15.873 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 326 (ObjNum 395 snap 44 seq_num 395) dirty exists 2024-04-20T02:53:15.873 INFO:tasks.rados.rados.0.smithi105.stdout:1937: left oid 6 (ObjNum 395 snap 44 seq_num 395) 2024-04-20T02:53:15.873 INFO:tasks.rados.rados.0.smithi105.stdout:1939: expect (ObjNum 359 snap 37 seq_num 359) 2024-04-20T02:53:15.915 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:15.916+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:15.961 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 154.22580222900001, 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:15.962 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:15.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:53:15.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:53:15.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:53:15.967 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 154.19451316799999, 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.968 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 154.18982271199999, 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:15.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:53:15.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:53:15.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:53:15.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:15.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.980 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:15.980 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:15.980 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 154.18306624300001, 2024-04-20T02:53:15.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:53:15.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:53:15.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:53:15.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:53:15.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 154.17325073200001, 2024-04-20T02:53:15.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:15.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:15.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:53:15.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 154.1722068, 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:15.992 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:53:15.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:53:15.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:15.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:53:15.999 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 154.17218564000001, 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:16.004 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.96461505900001, 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:53:16.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.95042041400001, 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:53:16.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:53:16.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.944453157, 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.016 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.93604715399999, 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:53:16.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:16.020 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.832415113, 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:53:16.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:53:16.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.60181880299999, 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:16.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:53:16.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:16.028 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.57487465400001, 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:53:16.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:53:16.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.54951638700001, 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:16.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:53:16.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.52137594300001, 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:53:16.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:53:16.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.040 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.484007338, 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:53:16.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:53:16.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:53:16.044 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.48371041499999, 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:16.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:53:16.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.048 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.478627297, 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:53:16.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.052 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 153.45162473799999, 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:53:16.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:53:16.056 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.057 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.057 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.057 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.057 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.057 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:16.058 INFO:tasks.rados.rados.0.smithi105.stdout:1941: expect (ObjNum 380 snap 41 seq_num 380) 2024-04-20T02:53:16.059 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:16.059 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:16.059+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:16.095 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 3, 6, 7, 5, 2, 4] 2024-04-20T02:53:16.096 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:53:16.096 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-20T02:53:16.096 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 1.000000 2024-04-20T02:53:16.096 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-20T02:53:16.145 INFO:tasks.rados.rados.0.smithi105.stdout:1929: done (7 left) 2024-04-20T02:53:16.145 INFO:tasks.rados.rados.0.smithi105.stdout:1934: done (6 left) 2024-04-20T02:53:16.145 INFO:tasks.rados.rados.0.smithi105.stdout:1935: done (5 left) 2024-04-20T02:53:16.146 INFO:tasks.rados.rados.0.smithi105.stdout:1936: done (4 left) 2024-04-20T02:53:16.146 INFO:tasks.rados.rados.0.smithi105.stdout:1937: done (3 left) 2024-04-20T02:53:16.146 INFO:tasks.rados.rados.0.smithi105.stdout:1939: done (2 left) 2024-04-20T02:53:16.146 INFO:tasks.rados.rados.0.smithi105.stdout:1941: done (1 left) 2024-04-20T02:53:16.146 INFO:tasks.rados.rados.0.smithi105.stdout:1942: done (0 left) 2024-04-20T02:53:16.146 INFO:tasks.rados.rados.0.smithi105.stdout:1943: write oid 26 current snap is 45 2024-04-20T02:53:16.146 INFO:tasks.rados.rados.0.smithi105.stdout:1943: seq_num 396 ranges {655572=648960,1922987=732585} 2024-04-20T02:53:16.149 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:53:16.156 INFO:tasks.rados.rados.0.smithi105.stdout:1943: writing smithi10548823-26 from 655572 to 1304532 tid 1 2024-04-20T02:53:16.158 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:16.159+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:16.162 INFO:tasks.rados.rados.0.smithi105.stdout:1943: writing smithi10548823-26 from 1922987 to 2655572 tid 2 2024-04-20T02:53:16.162 INFO:tasks.rados.rados.0.smithi105.stdout:1944: tier_evict oid 5 2024-04-20T02:53:16.162 INFO:tasks.rados.rados.0.smithi105.stdout:1944: tier_evict oid 5 snap -1 2024-04-20T02:53:16.162 INFO:tasks.rados.rados.0.smithi105.stdout:1945: snap_remove snap 44 2024-04-20T02:53:16.243 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:16.243 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:16.243 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:16.255 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:16.296 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:16.297+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:16.360 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:16.360+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:16.424 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:16.424 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:16.425 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:16.425 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:16.425 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:16.434 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:16.478 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:16.479+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:16.562 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:16.562+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:16.565 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:16.566 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:16.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:16.566 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:16.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.566 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:53:16.566 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:16.566 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 176.961313851, 2024-04-20T02:53:16.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:53:16.566 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.566 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:53:16.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 155.08414441100001, 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:16.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:16.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:53:16.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 154.72435875900001, 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:53:16.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:53:16.583 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:16.584 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 154.713995154, 2024-04-20T02:53:16.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:53:16.584 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.584 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.584 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:16.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:53:16.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:53:16.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:53:16.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:53:16.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:53:16.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:53:16.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:53:16.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 136.42435244999999, 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:16.589 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:53:16.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:53:16.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.595 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:16.596 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:16.596 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 136.41921533999999, 2024-04-20T02:53:16.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:53:16.596 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.596 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:16.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:16.596 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:16.596 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:53:16.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:53:16.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:16.602 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 136.41894232800001, 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:16.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:53:16.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:53:16.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:53:16.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:53:16.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:53:16.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:53:16.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:16.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:53:16.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:53:16.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:16.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:53:16.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:53:16.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.609 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:16.609 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:53:16.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:53:16.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:53:16.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 136.41891294300001, 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:16.611 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:53:16.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:53:16.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:53:16.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 113.73158564400001, 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.616 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.618 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:53:16.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:53:16.620 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.284105421999996, 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:16.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:53:16.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.266836987999994, 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:53:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.626 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.626 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:16.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:16.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:53:16.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:16.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:53:16.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:53:16.630 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 65.093691192999998, 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:16.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:53:16.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:53:16.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:53:16.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:53:16.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:53:16.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3881 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'232, mlcod=141'228)", 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.146568653000003, 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050172267999999999, 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.635 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418659+0000", 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419496+0000", 2024-04-20T02:53:16.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00083715799999999996 2024-04-20T02:53:16.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419513+0000", 2024-04-20T02:53:16.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7489000000000001e-05 2024-04-20T02:53:16.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419524+0000", 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0118e-05 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429881+0000", 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010357762 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429905+0000", 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3546000000000001e-05 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468790+0000", 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038884821 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468832+0000", 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2341e-05 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.639 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.640 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3898 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'133, mlcod=133'129)", 2024-04-20T02:53:16.640 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:16.640 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.094409681000002, 2024-04-20T02:53:16.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038834664999999997, 2024-04-20T02:53:16.640 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.640 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.640 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470817+0000", 2024-04-20T02:53:16.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471051+0000", 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023384600000000001 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471066+0000", 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4205e-05 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471074+0000", 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4090000000000003e-06 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472816+0000", 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017419429999999999 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472843+0000", 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7296000000000001e-05 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509592+0000", 2024-04-20T02:53:16.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036748496999999998 2024-04-20T02:53:16.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509653+0000", 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1707999999999999e-05 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3899 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'134, mlcod=133'129)", 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.077143245000002, 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038218993999999999, 2024-04-20T02:53:16.645 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488084+0000", 2024-04-20T02:53:16.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488262+0000", 2024-04-20T02:53:16.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017768600000000001 2024-04-20T02:53:16.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488274+0000", 2024-04-20T02:53:16.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2388000000000001e-05 2024-04-20T02:53:16.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488283+0000", 2024-04-20T02:53:16.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3840000000000003e-06 2024-04-20T02:53:16.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489094+0000", 2024-04-20T02:53:16.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081045100000000001 2024-04-20T02:53:16.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489119+0000", 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4751000000000001e-05 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526264+0000", 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037144912000000002 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526304+0000", 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0513000000000003e-05 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.650 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.651 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3891 4.7 e146/145 4:e2c0d65b:::smithi10548823-46:head v 146'105, mlcod=143'100)", 2024-04-20T02:53:16.651 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:16.651 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.076163149999999, 2024-04-20T02:53:16.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041132308999999999, 2024-04-20T02:53:16.651 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.651 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.651 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:16.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489072+0000", 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9539999999999998e-06 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489082+0000", 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0329000000000001e-05 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489089+0000", 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4959999999999999e-06 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489814+0000", 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072423699999999997 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489837+0000", 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3218999999999999e-05 2024-04-20T02:53:16.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530157+0000", 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040319762000000002 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530198+0000", 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0886000000000001e-05 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.7 146/145 [PushOp(4:e2c0d65b:::smithi10548823-46:43, version: 143'96, data_included: [426944~820638,1468778~227938], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:e2c0d65b:::smithi10548823-46:43@143'96, size: 3311012, copy_subset: [0~3311012], clone_subset: {}, snapset: 45=[]:{3f=[3f,39],43=[41],45=[45]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1696716, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.074951798000001, 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039876652999999998, 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490276+0000", 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490785+0000", 2024-04-20T02:53:16.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050941300000000001 2024-04-20T02:53:16.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490804+0000", 2024-04-20T02:53:16.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9080999999999999e-05 2024-04-20T02:53:16.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490823+0000", 2024-04-20T02:53:16.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.855e-05 2024-04-20T02:53:16.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498138+0000", 2024-04-20T02:53:16.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007315004 2024-04-20T02:53:16.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498153+0000", 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5515e-05 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530153+0000", 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031999845999999998 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4016 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'146, mlcod=147'143)", 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:16.660 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.302647552, 2024-04-20T02:53:16.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054600045999999999, 2024-04-20T02:53:16.661 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.661 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.661 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:16.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262580+0000", 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262733+0000", 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015289700000000001 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262746+0000", 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.375e-05 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262753+0000", 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6159999999999998e-06 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262817+0000", 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4202999999999995e-05 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262842+0000", 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5296e-05 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317137+0000", 2024-04-20T02:53:16.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054294664999999999 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317181+0000", 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.665 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4017 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'147, mlcod=147'143)", 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.287600865999998, 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039585437000000001, 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277626+0000", 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278058+0000", 2024-04-20T02:53:16.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043228299999999999 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278074+0000", 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5775000000000001e-05 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278083+0000", 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2620000000000004e-06 2024-04-20T02:53:16.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286849+0000", 2024-04-20T02:53:16.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087652210000000001 2024-04-20T02:53:16.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286881+0000", 2024-04-20T02:53:16.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1919999999999999e-05 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317188+0000", 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030307392999999998 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317213+0000", 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4953999999999999e-05 2024-04-20T02:53:16.670 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4340 4.8 e163/114 4:1a8c7fb2:::smithi10548823-47:head v 163'162, mlcod=159'158)", 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 13.652924480999999, 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.04004311, 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:16.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912303+0000", 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912722+0000", 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00041911999999999999 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912738+0000", 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6294e-05 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912748+0000", 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6059999999999992e-06 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912840+0000", 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2143000000000002e-05 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912867+0000", 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7121e-05 2024-04-20T02:53:16.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:16.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952298+0000", 2024-04-20T02:53:16.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039430999000000001 2024-04-20T02:53:16.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:16.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:16.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:16.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952347+0000", 2024-04-20T02:53:16.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8990999999999999e-05 2024-04-20T02:53:16.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.676 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:16.676 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:16.676 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:16.677 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:16.677+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:16.678 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:16.778 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:16.779+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:16.847 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:16.848 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:16.848 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:16.848 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:16.858 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:16.900 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:16.900+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:16.980 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:16.980+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:17.027 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:17.027 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:17.027 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:17.027 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:17.027 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:17.037 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:17.080 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:17.081+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:17.145 INFO:tasks.rados.rados.0.smithi105.stdout:1945: done (2 left) 2024-04-20T02:53:17.145 INFO:tasks.rados.rados.0.smithi105.stdout:1946: write oid 28 current snap is 45 2024-04-20T02:53:17.145 INFO:tasks.rados.rados.0.smithi105.stdout:1946: seq_num 397 ranges {672379=724209,1865116=608472} 2024-04-20T02:53:17.153 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-20T02:53:17.153 INFO:tasks.rados.rados.0.smithi105.stdout:1946: writing smithi10548823-28 from 672379 to 1396588 tid 1 2024-04-20T02:53:17.161 INFO:tasks.rados.rados.0.smithi105.stdout:1946: writing smithi10548823-28 from 1865116 to 2473588 tid 2 2024-04-20T02:53:17.161 INFO:tasks.rados.rados.0.smithi105.stdout:1947: read oid 34 snap -1 2024-04-20T02:53:17.161 INFO:tasks.rados.rados.0.smithi105.stdout:1947: expect (ObjNum 377 snap 40 seq_num 377) 2024-04-20T02:53:17.161 INFO:tasks.rados.rados.0.smithi105.stdout:1948: set_chunk oid 6 target oid 2024-04-20T02:53:17.169 INFO:tasks.rados.rados.0.smithi105.stdout:1948: set_chunk oid 6 offset: 2274304 length: 64512 target oid c658d2b09d5550659a4b66ff55bdc322997063f56aa604e3ad4863ad03cf3d1b offset: 0 snap -1 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1949: read oid 40 snap -1 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1949: expect (ObjNum 386 snap 41 seq_num 386) 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1950: read oid 8 snap -1 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1950: expect (ObjNum 336 snap 36 seq_num 336) 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1951: read oid 18 snap -1 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1951: expect (ObjNum 370 snap 37 seq_num 370) 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1952: read oid 12 snap -1 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1952: expect deleted 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1953: rollback oid 48 current snap is 45 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1953: done (9 left) 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1954: read oid 7 snap -1 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1954: expect (ObjNum 359 snap 37 seq_num 359) 2024-04-20T02:53:17.170 INFO:tasks.rados.rados.0.smithi105.stdout:1955: snap_create 2024-04-20T02:53:17.177 INFO:tasks.rados.rados.0.smithi105.stdout:1944: got (22) Invalid argument 2024-04-20T02:53:17.177 INFO:tasks.rados.rados.0.smithi105.stdout:1943: finishing write tid 1 to smithi10548823-26 2024-04-20T02:53:17.178 INFO:tasks.rados.rados.0.smithi105.stdout:1943: finishing write tid 2 to smithi10548823-26 2024-04-20T02:53:17.178 INFO:tasks.rados.rados.0.smithi105.stdout:1943: finishing write tid 3 to smithi10548823-26 2024-04-20T02:53:17.178 INFO:tasks.rados.rados.0.smithi105.stdout:1943: finishing write tid 4 to smithi10548823-26 2024-04-20T02:53:17.178 INFO:tasks.rados.rados.0.smithi105.stdout:1943: oid 26 updating version 0 to 470 2024-04-20T02:53:17.178 INFO:tasks.rados.rados.0.smithi105.stdout:1943: oid 26 updating version 470 to 472 2024-04-20T02:53:17.178 INFO:tasks.rados.rados.0.smithi105.stdout:1943: oid 26 version 472 is already newer than 471 2024-04-20T02:53:17.178 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 472 (ObjNum 396 snap 45 seq_num 396) dirty exists 2024-04-20T02:53:17.178 INFO:tasks.rados.rados.0.smithi105.stdout:1943: left oid 26 (ObjNum 396 snap 45 seq_num 396) 2024-04-20T02:53:17.178 INFO:tasks.rados.rados.0.smithi105.stdout:1947: expect (ObjNum 377 snap 40 seq_num 377) 2024-04-20T02:53:17.213 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:53:17.220 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:17.220+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:17.222 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:17.222 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:17.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:17.222 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:17.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.222 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:53:17.222 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:17.222 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 127.771676726, 2024-04-20T02:53:17.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:53:17.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 118.721493959, 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.228 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:53:17.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:53:17.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:53:17.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:53:17.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:53:17.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.231 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 118.72142851, 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:17.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 118.721403012, 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:17.236 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:53:17.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:53:17.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.240 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2524 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1723287~549075] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 88.933893529000002, 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054106970999999997, 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2524 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:17.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286419+0000", 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286924+0000", 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050494600000000004 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286944+0000", 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9833000000000001e-05 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286957+0000", 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2901000000000001e-05 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298074+0000", 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011117320999999999 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298151+0000", 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6909999999999994e-05 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298274+0000", 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012279700000000001 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326267+0000", 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027993284 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331226+0000", 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049590559999999999 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340442+0000", 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092155740000000007 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340488+0000", 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6208000000000002e-05 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340528+0000", 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0277000000000001e-05 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3637 4.1 e137/60 4:80c7a726:::smithi10548823-19:head v 137'124, mlcod=120'120)", 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.595905969999997, 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052640605, 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624408+0000", 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624537+0000", 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012880199999999999 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624548+0000", 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0216e-05 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624556+0000", 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6179999999999995e-06 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624603+0000", 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6601999999999999e-05 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624632+0000", 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9277000000000001e-05 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.676999+0000", 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052366639999999999 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.677049+0000", 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0822999999999998e-05 2024-04-20T02:53:17.250 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3880 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'231, mlcod=141'228)", 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.813410990000001, 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.061291667000000001, 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:17.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406903+0000", 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407359+0000", 2024-04-20T02:53:17.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00045623700000000002 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407370+0000", 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1052e-05 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407377+0000", 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1080000000000004e-06 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407429+0000", 2024-04-20T02:53:17.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2033999999999997e-05 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407458+0000", 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8050000000000001e-05 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468161+0000", 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060703628000000003 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468196+0000", 2024-04-20T02:53:17.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4344999999999997e-05 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3888 4.7 4:e2c0d65b:::smithi10548823-46:head [write 413538~536345,stat] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.794857862000001, 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.059175945000000001, 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.255 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3888 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425456+0000", 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426097+0000", 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00064115099999999996 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426113+0000", 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6118000000000001e-05 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426122+0000", 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.115e-06 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426179+0000", 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6815999999999999e-05 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426316+0000", 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000136606 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426499+0000", 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000183191 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443652+0000", 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017153471999999999 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484036+0000", 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040384097000000001 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484231+0000", 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000194811 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484363+0000", 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013146300000000001 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484426+0000", 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3563000000000001e-05 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484633+0000", 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020662299999999999 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3889 4.7 4:e2c0d65b:::smithi10548823-46:head [write 1395221~583781] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.787963886999997, 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.057918795000000002, 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.261 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3889 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432350+0000", 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433490+0000", 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001140606 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433506+0000", 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5828000000000002e-05 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433515+0000", 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8279999999999992e-06 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443692+0000", 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010177222 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443839+0000", 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014662 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.444072+0000", 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023362200000000001 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484114+0000", 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040041641000000003 2024-04-20T02:53:17.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484244+0000", 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012987200000000001 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484279+0000", 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5301e-05 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490055+0000", 2024-04-20T02:53:17.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057763459999999999 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490115+0000", 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9320999999999997e-05 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490270+0000", 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015498199999999999 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3890 4.7 4:e2c0d65b:::smithi10548823-46:head [write 2556103~658617] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.779949786000003, 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066878325000000002, 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3890 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440363+0000", 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441831+0000", 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001467632 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441842+0000", 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0540000000000001e-05 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441850+0000", 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7260000000000008e-06 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463536+0000", 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02168544 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463681+0000", 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000145086 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463941+0000", 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00025968600000000002 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484317+0000", 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020376452 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490318+0000", 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060012490000000002 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490398+0000", 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9598000000000006e-05 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506988+0000", 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016589706999999999 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507066+0000", 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8299000000000004e-05 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507243+0000", 2024-04-20T02:53:17.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000177503 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3891 4.7 4:e2c0d65b:::smithi10548823-46:head [setxattr _header (57) in=7b,truncate 3613538] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.778454277000002, 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088590031, 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.274 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3891 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441859+0000", 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441864+0000", 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8069999999999999e-06 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441869+0000", 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0989999999999998e-06 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441875+0000", 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5820000000000001e-06 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484675+0000", 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042799867999999998 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484817+0000", 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014236899999999999 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485048+0000", 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023057199999999999 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490359+0000", 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053116819999999999 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506963+0000", 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016603562999999998 2024-04-20T02:53:17.278 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507256+0000", 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00029288699999999998 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530316+0000", 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023060343000000001 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530356+0000", 2024-04-20T02:53:17.279 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9301999999999997e-05 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530451+0000", 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5037999999999994e-05 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3892 4.7 4:e2c0d65b:::smithi10548823-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.778434589, 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088912482000000001, 2024-04-20T02:53:17.280 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3892 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.281 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441883+0000", 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9579999999999999e-06 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.282 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441889+0000", 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7499999999999997e-06 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441895+0000", 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485431+0000", 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043536321000000003 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.283 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485537+0000", 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010612599999999999 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530489+0000", 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044951996000000001 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530598+0000", 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108937 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.284 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530793+0000", 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019441 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3898 4.8 4:1c68e542:::smithi10548823-8:head [write 447152~403196] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.754091506000002, 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.056917473000000003, 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.285 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3898 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:17.286 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466222+0000", 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466722+0000", 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00049941299999999998 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466734+0000", 2024-04-20T02:53:17.287 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2860999999999999e-05 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466743+0000", 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7159999999999992e-06 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466809+0000", 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6165000000000001e-05 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466928+0000", 2024-04-20T02:53:17.288 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011869199999999999 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.467158+0000", 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000230251 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498338+0000", 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.03117996 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.289 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.522958+0000", 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024619318000000001 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523002+0000", 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4675999999999997e-05 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523082+0000", 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9803999999999997e-05 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.290 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523141+0000", 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8767000000000002e-05 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3899 4.8 4:1c68e542:::smithi10548823-8:head [write 1512185~587450] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.746163738999996, 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060518556000000001, 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.291 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3899 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.292 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474149+0000", 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474686+0000", 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053696199999999999 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474703+0000", 2024-04-20T02:53:17.293 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.615e-05 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474713+0000", 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0829000000000001e-05 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478533+0000", 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003819687 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.294 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478640+0000", 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106974 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478766+0000", 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125859 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523803+0000", 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045036616000000002 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.295 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526432+0000", 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026291489999999999 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534589+0000", 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081572209999999992 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534623+0000", 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.296 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534670+0000", 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6968999999999997e-05 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4016 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 665643~697786] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.958994973999999, 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060676251, 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.297 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4016 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.298 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261318+0000", 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262081+0000", 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076285299999999997 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262095+0000", 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3986e-05 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.299 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262118+0000", 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2407000000000001e-05 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262164+0000", 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6248999999999997e-05 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262238+0000", 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4202000000000005e-05 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.300 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262363+0000", 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012437799999999999 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314944+0000", 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052581687000000002 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317407+0000", 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024628300000000001 2024-04-20T02:53:17.301 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321887+0000", 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0044803020000000002 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321941+0000", 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3769e-05 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321996+0000", 2024-04-20T02:53:17.302 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4988000000000003e-05 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4017 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 2039745~753122] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 28.951365175999999, 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053146383999999998, 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.303 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4017 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.304 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268948+0000", 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269678+0000", 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072975799999999995 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269693+0000", 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.453e-05 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.305 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269704+0000", 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1623e-05 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277190+0000", 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074859849999999997 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277276+0000", 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5809999999999993e-05 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.306 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277405+0000", 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012914600000000001 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314952+0000", 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037546931999999998 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317441+0000", 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024889840000000001 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.307 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322031+0000", 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004589465 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322059+0000", 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322096+0000", 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591999999999998e-05 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.308 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4074 4.11 4:8b91c759:::smithi10548823-5:head [write 1591248~408851] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 25.078397787, 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053876146, 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4074 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.309 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141916+0000", 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.310 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142740+0000", 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000823616 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142750+0000", 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6150000000000003e-06 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142756+0000", 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9419999999999998e-06 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.311 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2061999999999997e-05 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142815+0000", 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5050000000000004e-06 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174069+0000", 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031253933999999997 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.312 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174193+0000", 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012386 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174438+0000", 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00024561400000000001 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187472+0000", 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013033375999999999 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.313 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.188278+0000", 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00080595500000000002 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195551+0000", 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072733340000000002 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195631+0000", 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9882999999999994e-05 2024-04-20T02:53:17.314 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195793+0000", 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000162223 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4075 4.11 4:8b91c759:::smithi10548823-5:head [setxattr _header (56) in=7b,truncate 2000099] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 25.077546855000001, 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084064655000000002, 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.315 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4075 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.316 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142767+0000", 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142778+0000", 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0709e-05 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.317 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142783+0000", 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4539999999999998e-06 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142788+0000", 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2850000000000002e-06 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142845+0000", 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6919000000000002e-05 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.318 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142856+0000", 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0936e-05 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186635+0000", 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043778897999999997 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186756+0000", 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121087 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.319 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186967+0000", 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021064599999999999 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.189477+0000", 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025104929999999999 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.190395+0000", 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00091717399999999998 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.320 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226595+0000", 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036200195999999997 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226661+0000", 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6465999999999997e-05 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226833+0000", 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000171471 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.321 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4076 4.11 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 25.077505849000001, 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084363853000000003, 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4076 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.322 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142808+0000", 2024-04-20T02:53:17.323 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142811+0000", 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9229999999999998e-06 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142818+0000", 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4420000000000003e-06 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142823+0000", 2024-04-20T02:53:17.324 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5199999999999997e-06 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142875+0000", 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1759999999999999e-05 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142880+0000", 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6639999999999997e-06 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.325 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187624+0000", 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044743949999999998 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187726+0000", 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0001022 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226882+0000", 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039155862 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.326 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227003+0000", 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120715 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227173+0000", 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017032899999999999 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4183 3.2 e154/49 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head v 154'234, mlcod=154'232)", 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:17.327 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 23.567376916000001, 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.063121930000000007, 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:17.328 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652936+0000", 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657053+0000", 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041172520000000001 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657067+0000", 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3865e-05 2024-04-20T02:53:17.329 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657076+0000", 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5390000000000007e-06 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657121+0000", 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5114000000000002e-05 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657138+0000", 2024-04-20T02:53:17.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7600000000000001e-05 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716021+0000", 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.058882663000000002 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716060+0000", 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8826999999999999e-05 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:17.331 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:17.332 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:17.334 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:17.334 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:17.334+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:17.334 INFO:tasks.rados.rados.0.smithi105.stdout:1954: expect (ObjNum 359 snap 37 seq_num 359) 2024-04-20T02:53:17.367 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:17.386 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:17.406 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:17.426 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:17.434 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:17.434+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:17.438 INFO:tasks.rados.rados.0.smithi105.stdout:1951: expect (ObjNum 370 snap 37 seq_num 370) 2024-04-20T02:53:17.445 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:17.465 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:17.485 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:17.498 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:17.498 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:17.505 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:17.506 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:17.506 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:17.506 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:17.506 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:17.506 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:17.507 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:17.507 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:17.507 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:17.509 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:17.560 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:17.559+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:17.562 INFO:tasks.rados.rados.0.smithi105.stdout:1949: expect (ObjNum 386 snap 41 seq_num 386) 2024-04-20T02:53:17.636 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:17.636+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:17.687 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:17.687 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:17.697 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:17.739 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:17.739+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:17.769 INFO:tasks.rados.rados.0.smithi105.stdout:1948: finishing set_chunk to oid 6 2024-04-20T02:53:17.769 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 6 v 326 (ObjNum 395 snap 44 seq_num 395) dirty exists 2024-04-20T02:53:17.769 INFO:tasks.rados.rados.0.smithi105.stdout:1950: expect (ObjNum 336 snap 36 seq_num 336) 2024-04-20T02:53:17.838 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:17.838+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:17.868 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:17.868 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:17.878 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:17.938 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:17.939+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:17.969 INFO:tasks.rados.rados.0.smithi105.stdout:1946: finishing write tid 1 to smithi10548823-28 2024-04-20T02:53:17.969 INFO:tasks.rados.rados.0.smithi105.stdout:1946: finishing write tid 2 to smithi10548823-28 2024-04-20T02:53:17.969 INFO:tasks.rados.rados.0.smithi105.stdout:1946: finishing write tid 3 to smithi10548823-28 2024-04-20T02:53:17.969 INFO:tasks.rados.rados.0.smithi105.stdout:1946: finishing write tid 4 to smithi10548823-28 2024-04-20T02:53:17.970 INFO:tasks.rados.rados.0.smithi105.stdout:1946: oid 28 updating version 0 to 232 2024-04-20T02:53:17.970 INFO:tasks.rados.rados.0.smithi105.stdout:1946: oid 28 version 232 is already newer than 230 2024-04-20T02:53:17.970 INFO:tasks.rados.rados.0.smithi105.stdout:1946: oid 28 version 232 is already newer than 231 2024-04-20T02:53:17.970 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 28 v 232 (ObjNum 397 snap 45 seq_num 397) dirty exists 2024-04-20T02:53:17.970 INFO:tasks.rados.rados.0.smithi105.stdout:1946: left oid 28 (ObjNum 397 snap 45 seq_num 397) 2024-04-20T02:53:18.008 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:18.008 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:18.019 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:18.062 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:18.062+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:18.141 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:18.140+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:18.144 INFO:tasks.rados.rados.0.smithi105.stdout:1943: done (10 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1944: done (9 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1946: done (8 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1947: done (7 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1948: done (6 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1949: done (5 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1950: done (4 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1951: done (3 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1952: done (2 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1954: done (1 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1955: done (0 left) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1956: read oid 41 snap -1 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1956: expect (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:53:18.145 INFO:tasks.rados.rados.0.smithi105.stdout:1957: delete oid 33 current snap is 46 2024-04-20T02:53:18.154 INFO:tasks.rados.rados.0.smithi105.stdout:1957: done (1 left) 2024-04-20T02:53:18.154 INFO:tasks.rados.rados.0.smithi105.stdout:1958: read oid 26 snap 45 2024-04-20T02:53:18.154 INFO:tasks.rados.rados.0.smithi105.stdout:1958: expect (ObjNum 396 snap 45 seq_num 396) 2024-04-20T02:53:18.154 INFO:tasks.rados.rados.0.smithi105.stdout:1959: delete oid 39 current snap is 46 2024-04-20T02:53:18.154 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:18.154 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:18.162 INFO:tasks.rados.rados.0.smithi105.stdout:1958: expect (ObjNum 396 snap 45 seq_num 396) 2024-04-20T02:53:18.165 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:18.165 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:53:18.242 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:18.242+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:18.295 INFO:tasks.rados.rados.0.smithi105.stdout:1958: done (2 left) 2024-04-20T02:53:18.296 INFO:tasks.rados.rados.0.smithi105.stdout:1959: done (1 left) 2024-04-20T02:53:18.296 INFO:tasks.rados.rados.0.smithi105.stdout:1960: write oid 15 current snap is 46 2024-04-20T02:53:18.296 INFO:tasks.rados.rados.0.smithi105.stdout:1960: seq_num 400 ranges {722800=549956,2040562=682238} 2024-04-20T02:53:18.302 INFO:tasks.rados.rados.0.smithi105.stdout:1960: writing smithi10548823-15 from 722800 to 1272756 tid 1 2024-04-20T02:53:18.310 INFO:tasks.rados.rados.0.smithi105.stdout:1960: writing smithi10548823-15 from 2040562 to 2722800 tid 2 2024-04-20T02:53:18.310 INFO:tasks.rados.rados.0.smithi105.stdout:1961: delete oid 45 current snap is 46 2024-04-20T02:53:18.314 INFO:tasks.rados.rados.0.smithi105.stdout:1961: done (2 left) 2024-04-20T02:53:18.314 INFO:tasks.rados.rados.0.smithi105.stdout:1962: delete oid 32 current snap is 46 2024-04-20T02:53:18.318 INFO:tasks.rados.rados.0.smithi105.stdout:1962: done (2 left) 2024-04-20T02:53:18.318 INFO:tasks.rados.rados.0.smithi105.stdout:1963: write (excl) oid 20 current snap is 46 2024-04-20T02:53:18.318 INFO:tasks.rados.rados.0.smithi105.stdout:1963: seq_num 403 ranges {773221=492056,1816008=503126} 2024-04-20T02:53:18.325 INFO:tasks.rados.rados.0.smithi105.stdout:1963: writing smithi10548823-20 from 773221 to 1265277 tid 1 2024-04-20T02:53:18.332 INFO:tasks.rados.rados.0.smithi105.stdout:1963: writing smithi10548823-20 from 1816008 to 2319134 tid 2 2024-04-20T02:53:18.332 INFO:tasks.rados.rados.0.smithi105.stdout:1964: read oid 17 snap -1 2024-04-20T02:53:18.332 INFO:tasks.rados.rados.0.smithi105.stdout:1964: expect (ObjNum 340 snap 36 seq_num 340) 2024-04-20T02:53:18.332 INFO:tasks.rados.rados.0.smithi105.stdout:1965: tier_flush oid 13 2024-04-20T02:53:18.333 INFO:tasks.rados.rados.0.smithi105.stdout:1965: tier_flush oid 13 snap -1 2024-04-20T02:53:18.333 INFO:tasks.rados.rados.0.smithi105.stdout:1966: delete oid 30 current snap is 46 2024-04-20T02:53:18.336 INFO:tasks.rados.rados.0.smithi105.stdout:1960: finishing write tid 1 to smithi10548823-15 2024-04-20T02:53:18.336 INFO:tasks.rados.rados.0.smithi105.stdout:1960: finishing write tid 2 to smithi10548823-15 2024-04-20T02:53:18.336 INFO:tasks.rados.rados.0.smithi105.stdout:1965: got (16) Device or resource busy 2024-04-20T02:53:18.336 INFO:tasks.rados.rados.0.smithi105.stdout:check_oldest_snap_flushed oid 13 v 422 (ObjNum 345 snap 37 seq_num 345) unflushed 45 2024-04-20T02:53:18.339 INFO:tasks.rados.rados.0.smithi105.stdout:1960: finishing write tid 3 to smithi10548823-15 2024-04-20T02:53:18.340 INFO:tasks.rados.rados.0.smithi105.stdout:1960: finishing write tid 4 to smithi10548823-15 2024-04-20T02:53:18.340 INFO:tasks.rados.rados.0.smithi105.stdout:1960: oid 15 updating version 0 to 475 2024-04-20T02:53:18.340 INFO:tasks.rados.rados.0.smithi105.stdout:1960: oid 15 version 475 is already newer than 473 2024-04-20T02:53:18.340 INFO:tasks.rados.rados.0.smithi105.stdout:1960: oid 15 version 475 is already newer than 474 2024-04-20T02:53:18.340 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 15 v 475 (ObjNum 400 snap 46 seq_num 400) dirty exists 2024-04-20T02:53:18.340 INFO:tasks.rados.rados.0.smithi105.stdout:1960: left oid 15 (ObjNum 400 snap 46 seq_num 400) 2024-04-20T02:53:18.340 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:18.340 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:18.341 INFO:tasks.rados.rados.0.smithi105.stdout:1960: done (5 left) 2024-04-20T02:53:18.341 INFO:tasks.rados.rados.0.smithi105.stdout:1965: done (4 left) 2024-04-20T02:53:18.341 INFO:tasks.rados.rados.0.smithi105.stdout:1966: done (3 left) 2024-04-20T02:53:18.341 INFO:tasks.rados.rados.0.smithi105.stdout:1967: read oid 25 snap -1 2024-04-20T02:53:18.341 INFO:tasks.rados.rados.0.smithi105.stdout:1967: expect (ObjNum 383 snap 41 seq_num 383) 2024-04-20T02:53:18.341 INFO:tasks.rados.rados.0.smithi105.stdout:1968: read oid 6 snap 45 2024-04-20T02:53:18.342 INFO:tasks.rados.rados.0.smithi105.stdout:1968: expect (ObjNum 395 snap 44 seq_num 395) 2024-04-20T02:53:18.345 INFO:tasks.rados.rados.0.smithi105.stdout:1969: read oid 23 snap 45 2024-04-20T02:53:18.345 INFO:tasks.rados.rados.0.smithi105.stdout:1969: expect (ObjNum 373 snap 38 seq_num 373) 2024-04-20T02:53:18.345 INFO:tasks.rados.rados.0.smithi105.stdout:1970: set_chunk oid 45 target oid 2024-04-20T02:53:18.346 INFO:tasks.rados.rados.0.smithi105.stdout:1970: set_chunk oid 45 offset: 0 length: 4000000 target oid smithi10548823- offset: 0 snap -1 2024-04-20T02:53:18.346 INFO:tasks.rados.rados.0.smithi105.stdout:1971: write oid 7 current snap is 46 2024-04-20T02:53:18.346 INFO:tasks.rados.rados.0.smithi105.stdout:1971: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-04-20T02:53:18.347 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:18.346+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:18.348 INFO:tasks.rados.rados.0.smithi105.stdout:1971: writing smithi10548823-7 from 406835 to 1049389 tid 1 2024-04-20T02:53:18.353 INFO:tasks.rados.rados.0.smithi105.stdout:1971: writing smithi10548823-7 from 1532972 to 2039767 tid 2 2024-04-20T02:53:18.355 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:18.358 INFO:tasks.rados.rados.0.smithi105.stdout:1971: writing smithi10548823-7 from 2772312 to 2806835 tid 3 2024-04-20T02:53:18.358 INFO:tasks.rados.rados.0.smithi105.stdout:1972: read oid 3 snap -1 2024-04-20T02:53:18.358 INFO:tasks.rados.rados.0.smithi105.stdout:1972: expect (ObjNum 372 snap 38 seq_num 372) 2024-04-20T02:53:18.358 INFO:tasks.rados.rados.0.smithi105.stdout:1973: write oid 27 current snap is 46 2024-04-20T02:53:18.358 INFO:tasks.rados.rados.0.smithi105.stdout:1973: seq_num 406 ranges {423642=717803,1875101=766806} 2024-04-20T02:53:18.364 INFO:tasks.rados.rados.0.smithi105.stdout:1973: writing smithi10548823-27 from 423642 to 1141445 tid 1 2024-04-20T02:53:18.374 INFO:tasks.rados.rados.0.smithi105.stdout:1973: writing smithi10548823-27 from 1875101 to 2641907 tid 2 2024-04-20T02:53:18.374 INFO:tasks.rados.rados.0.smithi105.stdout:1974: set_chunk oid 42 target oid 2024-04-20T02:53:18.398 INFO:tasks.rados.rados.0.smithi105.stdout:1974: set_chunk oid 42 offset: 2092544 length: 35840 target oid 9992b2b32c8e665f80fa78a30fa5718aee332aef6cdc4b5e5f44712d6bfcdae6 offset: 0 snap -1 2024-04-20T02:53:18.399 INFO:tasks.rados.rados.0.smithi105.stdout:1975: read oid 15 snap -1 2024-04-20T02:53:18.399 INFO:tasks.rados.rados.0.smithi105.stdout:1975: expect (ObjNum 400 snap 46 seq_num 400) 2024-04-20T02:53:18.399 INFO:tasks.rados.rados.0.smithi105.stdout:1976: read oid 28 snap -1 2024-04-20T02:53:18.399 INFO:tasks.rados.rados.0.smithi105.stdout:1976: expect (ObjNum 397 snap 45 seq_num 397) 2024-04-20T02:53:18.399 INFO:tasks.rados.rados.0.smithi105.stdout:1977: rollback oid 19 current snap is 46 2024-04-20T02:53:18.399 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 19 to 45 2024-04-20T02:53:18.399 INFO:tasks.rados.rados.0.smithi105.stdout:1978: set_chunk oid 36 target oid 2024-04-20T02:53:18.442 INFO:tasks.rados.rados.0.smithi105.stdout:1978: set_chunk oid 36 offset: 732672 length: 1242624 target oid 1493d055885f5e337708a27dd5ebc51bd6b01b5af88d062fd6aaa639eb2f6cf7 offset: 0 snap -1 2024-04-20T02:53:18.442 INFO:tasks.rados.rados.0.smithi105.stdout:1979: write (excl) oid 46 current snap is 46 2024-04-20T02:53:18.442 INFO:tasks.rados.rados.0.smithi105.stdout:1970: finishing set_chunk to oid 45 2024-04-20T02:53:18.443 INFO:tasks.rados.rados.0.smithi105.stdout:1970: got expected ERANGE (src dne) 2024-04-20T02:53:18.443 INFO:tasks.rados.rados.0.smithi105.stdout:1979: seq_num 407 ranges {440449=793052,1933583=510464} 2024-04-20T02:53:18.443 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:18.444+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:18.449 INFO:tasks.rados.rados.0.smithi105.stdout:1979: writing smithi10548823-46 from 440449 to 1233501 tid 1 2024-04-20T02:53:18.457 INFO:tasks.rados.rados.0.smithi105.stdout:1979: writing smithi10548823-46 from 1933583 to 2444047 tid 2 2024-04-20T02:53:18.457 INFO:tasks.rados.rados.0.smithi105.stdout:1970: done (15 left) 2024-04-20T02:53:18.457 INFO:tasks.rados.rados.0.smithi105.stdout:1980: read oid 48 snap -1 2024-04-20T02:53:18.457 INFO:tasks.rados.rados.0.smithi105.stdout:1980: expect (ObjNum 376 snap 40 seq_num 376) 2024-04-20T02:53:18.457 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:53:18.457 INFO:tasks.rados.rados.0.smithi105.stdout:1969: expect (ObjNum 373 snap 38 seq_num 373) 2024-04-20T02:53:18.516 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:18.516 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:18.527 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:18.569 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:18.569+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:18.614 INFO:tasks.rados.rados.0.smithi105.stdout:1964: expect (ObjNum 340 snap 36 seq_num 340) 2024-04-20T02:53:18.645 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:18.645+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:18.698 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:18.698 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:18.709 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:18.743 INFO:tasks.rados.rados.0.smithi105.stdout:1972: expect (ObjNum 372 snap 38 seq_num 372) 2024-04-20T02:53:18.746 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:18.745+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:18.818 INFO:tasks.rados.rados.0.smithi105.stdout:1967: expect (ObjNum 383 snap 41 seq_num 383) 2024-04-20T02:53:18.839 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:18.839 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:18.846 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:18.845+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:18.850 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:18.946 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:18.946+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:19.020 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:19.021 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:19.031 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:1963: finishing write tid 1 to smithi10548823-20 2024-04-20T02:53:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:1963: finishing write tid 2 to smithi10548823-20 2024-04-20T02:53:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:1963: finishing write tid 3 to smithi10548823-20 2024-04-20T02:53:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:1963: finishing write tid 4 to smithi10548823-20 2024-04-20T02:53:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:1963: oid 20 updating version 0 to 341 2024-04-20T02:53:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:1963: oid 20 version 341 is already newer than 340 2024-04-20T02:53:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:1963: oid 20 version 341 is already newer than 339 2024-04-20T02:53:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 341 (ObjNum 403 snap 46 seq_num 403) dirty exists 2024-04-20T02:53:19.039 INFO:tasks.rados.rados.0.smithi105.stdout:1963: left oid 20 (ObjNum 403 snap 46 seq_num 403) 2024-04-20T02:53:19.040 INFO:tasks.rados.rados.0.smithi105.stdout:1968: expect (ObjNum 395 snap 44 seq_num 395) 2024-04-20T02:53:19.047 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:19.047+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:19.131 INFO:tasks.rados.rados.0.smithi105.stdout:1975: expect (ObjNum 400 snap 46 seq_num 400) 2024-04-20T02:53:19.148 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:19.149+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:19.150 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:53:19.178 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:19.178 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:19.187 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:19.228 INFO:tasks.rados.rados.0.smithi105.stdout:1976: expect (ObjNum 397 snap 45 seq_num 397) 2024-04-20T02:53:19.250 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:19.249+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:19.351 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:19.351+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:19.358 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:19.358 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:19.369 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:19.451 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:19.451+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:19.454 INFO:tasks.rados.rados.0.smithi105.stdout:1977: finishing rollback tid 0 to smithi10548823-19 2024-04-20T02:53:19.454 INFO:tasks.rados.rados.0.smithi105.stdout:1974: finishing set_chunk to oid 42 2024-04-20T02:53:19.454 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 42 v 96 (ObjNum 375 snap 39 seq_num 375) dirty exists 2024-04-20T02:53:19.454 INFO:tasks.rados.rados.0.smithi105.stdout:1977: finishing rollback tid 1 to smithi10548823-19 2024-04-20T02:53:19.454 INFO:tasks.rados.rados.0.smithi105.stdout:1977: finishing rollback tid 2 to smithi10548823-19 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 19 v 137 (ObjNum 311 snap 33 seq_num 311) dirty exists 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:1973: finishing write tid 1 to smithi10548823-27 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:1973: finishing write tid 2 to smithi10548823-27 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:1973: finishing write tid 3 to smithi10548823-27 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:1973: finishing write tid 4 to smithi10548823-27 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:1973: oid 27 updating version 0 to 344 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:1973: oid 27 version 344 is already newer than 342 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:1973: oid 27 version 344 is already newer than 343 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 27 v 344 (ObjNum 406 snap 46 seq_num 406) dirty exists 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:1973: left oid 27 (ObjNum 406 snap 46 seq_num 406) 2024-04-20T02:53:19.455 INFO:tasks.rados.rados.0.smithi105.stdout:1963: done (15 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1964: done (14 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1967: done (13 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1968: done (12 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1969: done (11 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1972: done (10 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1973: done (9 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1974: done (8 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1975: done (7 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1976: done (6 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1977: done (5 left) 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1981: write oid 26 current snap is 46 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:1978: finishing set_chunk to oid 36 2024-04-20T02:53:19.456 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 36 v 452 (ObjNum 385 snap 41 seq_num 385) dirty exists 2024-04-20T02:53:19.457 INFO:tasks.rados.rados.0.smithi105.stdout:1980: expect (ObjNum 376 snap 40 seq_num 376) 2024-04-20T02:53:19.499 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:19.500 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:19.510 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:19.518 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:19.535 INFO:tasks.rados.rados.0.smithi105.stdout:1979: finishing write tid 1 to smithi10548823-46 2024-04-20T02:53:19.535 INFO:tasks.rados.rados.0.smithi105.stdout:1979: finishing write tid 2 to smithi10548823-46 2024-04-20T02:53:19.535 INFO:tasks.rados.rados.0.smithi105.stdout:1979: finishing write tid 3 to smithi10548823-46 2024-04-20T02:53:19.535 INFO:tasks.rados.rados.0.smithi105.stdout:1979: finishing write tid 4 to smithi10548823-46 2024-04-20T02:53:19.535 INFO:tasks.rados.rados.0.smithi105.stdout:1979: oid 46 updating version 0 to 126 2024-04-20T02:53:19.536 INFO:tasks.rados.rados.0.smithi105.stdout:1979: oid 46 updating version 126 to 127 2024-04-20T02:53:19.536 INFO:tasks.rados.rados.0.smithi105.stdout:1979: oid 46 updating version 127 to 128 2024-04-20T02:53:19.536 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 46 v 128 (ObjNum 407 snap 46 seq_num 407) dirty exists 2024-04-20T02:53:19.536 INFO:tasks.rados.rados.0.smithi105.stdout:1979: left oid 46 (ObjNum 407 snap 46 seq_num 407) 2024-04-20T02:53:19.536 INFO:tasks.rados.rados.0.smithi105.stdout:1981: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-04-20T02:53:19.540 INFO:tasks.rados.rados.0.smithi105.stdout:1981: writing smithi10548823-26 from 457256 to 925557 tid 1 2024-04-20T02:53:19.549 INFO:tasks.rados.rados.0.smithi105.stdout:1981: writing smithi10548823-26 from 1592065 to 2362540 tid 2 2024-04-20T02:53:19.552 INFO:tasks.rados.rados.0.smithi105.stdout:1981: writing smithi10548823-26 from 2854581 to 2857256 tid 3 2024-04-20T02:53:19.552 INFO:tasks.rados.rados.0.smithi105.stdout:1978: done (5 left) 2024-04-20T02:53:19.552 INFO:tasks.rados.rados.0.smithi105.stdout:1979: done (4 left) 2024-04-20T02:53:19.552 INFO:tasks.rados.rados.0.smithi105.stdout:1980: done (3 left) 2024-04-20T02:53:19.552 INFO:tasks.rados.rados.0.smithi105.stdout:1982: delete oid 11 current snap is 46 2024-04-20T02:53:19.553 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:19.552+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:19.557 INFO:tasks.rados.rados.0.smithi105.stdout:1982: done (3 left) 2024-04-20T02:53:19.557 INFO:tasks.rados.rados.0.smithi105.stdout:1983: write oid 1 current snap is 46 2024-04-20T02:53:19.557 INFO:tasks.rados.rados.0.smithi105.stdout:1983: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-04-20T02:53:19.564 INFO:tasks.rados.rados.0.smithi105.stdout:1983: writing smithi10548823-1 from 490870 to 1109669 tid 1 2024-04-20T02:53:19.572 INFO:tasks.rados.rados.0.smithi105.stdout:1983: writing smithi10548823-1 from 1592676 to 2366820 tid 2 2024-04-20T02:53:19.576 INFO:tasks.rados.rados.0.smithi105.stdout:1983: writing smithi10548823-1 from 2793074 to 2890870 tid 3 2024-04-20T02:53:19.576 INFO:tasks.rados.rados.0.smithi105.stdout:1984: read oid 36 snap 45 2024-04-20T02:53:19.576 INFO:tasks.rados.rados.0.smithi105.stdout:1984: expect (ObjNum 385 snap 41 seq_num 385) 2024-04-20T02:53:19.576 INFO:tasks.rados.rados.0.smithi105.stdout:1985: write oid 14 current snap is 46 2024-04-20T02:53:19.576 INFO:tasks.rados.rados.0.smithi105.stdout:1985: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-04-20T02:53:19.580 INFO:tasks.rados.rados.0.smithi105.stdout:1985: writing smithi10548823-14 from 507677 to 918078 tid 1 2024-04-20T02:53:19.586 INFO:tasks.rados.rados.0.smithi105.stdout:1985: writing smithi10548823-14 from 1367511 to 1885313 tid 2 2024-04-20T02:53:19.593 INFO:tasks.rados.rados.0.smithi105.stdout:1985: writing smithi10548823-14 from 2420497 to 2907677 tid 3 2024-04-20T02:53:19.593 INFO:tasks.rados.rados.0.smithi105.stdout:1986: read oid 40 snap 45 2024-04-20T02:53:19.593 INFO:tasks.rados.rados.0.smithi105.stdout:1986: expect (ObjNum 386 snap 41 seq_num 386) 2024-04-20T02:53:19.593 INFO:tasks.rados.rados.0.smithi105.stdout:1987: write (excl) oid 13 current snap is 46 2024-04-20T02:53:19.593 INFO:tasks.rados.rados.0.smithi105.stdout:1981: finishing write tid 1 to smithi10548823-26 2024-04-20T02:53:19.593 INFO:tasks.rados.rados.0.smithi105.stdout:1987: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-04-20T02:53:19.598 INFO:tasks.rados.rados.0.smithi105.stdout:1987: writing smithi10548823-13 from 524484 to 1010134 tid 1 2024-04-20T02:53:19.608 INFO:tasks.rados.rados.0.smithi105.stdout:1987: writing smithi10548823-13 from 1425993 to 2087453 tid 2 2024-04-20T02:53:19.612 INFO:tasks.rados.rados.0.smithi105.stdout:1987: writing smithi10548823-13 from 2847920 to 2924484 tid 3 2024-04-20T02:53:19.612 INFO:tasks.rados.rados.0.smithi105.stdout:1988: read oid 18 snap -1 2024-04-20T02:53:19.612 INFO:tasks.rados.rados.0.smithi105.stdout:1988: expect (ObjNum 370 snap 37 seq_num 370) 2024-04-20T02:53:19.612 INFO:tasks.rados.rados.0.smithi105.stdout:1989: read oid 35 snap 45 2024-04-20T02:53:19.612 INFO:tasks.rados.rados.0.smithi105.stdout:1989: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:53:19.612 INFO:tasks.rados.rados.0.smithi105.stdout:1990: snap_create 2024-04-20T02:53:19.612 INFO:tasks.rados.rados.0.smithi105.stdout:1981: finishing write tid 2 to smithi10548823-26 2024-04-20T02:53:19.612 INFO:tasks.rados.rados.0.smithi105.stdout:1981: finishing write tid 3 to smithi10548823-26 2024-04-20T02:53:19.613 INFO:tasks.rados.rados.0.smithi105.stdout:1984: expect (ObjNum 385 snap 41 seq_num 385) 2024-04-20T02:53:19.653 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:19.652+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:19.675 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:19.675 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:19.687 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:19.726 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:19.745 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:19.753 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:19.753+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:19.766 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:19.809 INFO:tasks.rados.rados.0.smithi105.stdout:1981: finishing write tid 4 to smithi10548823-26 2024-04-20T02:53:19.809 INFO:tasks.rados.rados.0.smithi105.stdout:1981: finishing write tid 5 to smithi10548823-26 2024-04-20T02:53:19.810 INFO:tasks.rados.rados.0.smithi105.stdout:1981: oid 26 updating version 0 to 484 2024-04-20T02:53:19.810 INFO:tasks.rados.rados.0.smithi105.stdout:1981: oid 26 version 484 is already newer than 482 2024-04-20T02:53:19.810 INFO:tasks.rados.rados.0.smithi105.stdout:1981: oid 26 version 484 is already newer than 483 2024-04-20T02:53:19.810 INFO:tasks.rados.rados.0.smithi105.stdout:1981: oid 26 updating version 484 to 485 2024-04-20T02:53:19.810 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 485 (ObjNum 408 snap 46 seq_num 408) dirty exists 2024-04-20T02:53:19.810 INFO:tasks.rados.rados.0.smithi105.stdout:1981: left oid 26 (ObjNum 408 snap 46 seq_num 408) 2024-04-20T02:53:19.810 INFO:tasks.rados.rados.0.smithi105.stdout:1986: expect (ObjNum 386 snap 41 seq_num 386) 2024-04-20T02:53:19.854 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:19.854+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:19.855 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:19.855 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:19.866 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:19.955 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:19.955+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:19.986 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:20.004 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:20.005 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:20.005 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:20.011 INFO:tasks.rados.rados.0.smithi105.stdout:1985: finishing write tid 1 to smithi10548823-14 2024-04-20T02:53:20.011 INFO:tasks.rados.rados.0.smithi105.stdout:1988: expect (ObjNum 370 snap 37 seq_num 370) 2024-04-20T02:53:20.015 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:20.024 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:20.043 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:20.059 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:20.059+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:20.062 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:20.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:20.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:20.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:20.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:20.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:20.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:20.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:20.064 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:20.134 INFO:tasks.rados.rados.0.smithi105.stdout:1989: expect (ObjNum 124 snap 9 seq_num 124) 2024-04-20T02:53:20.144 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:20.144 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:20.144 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:20.144 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:20.144 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:20.157 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:20.158 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:20.157+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:20.203 INFO:tasks.rados.rados.0.smithi105.stdout:1985: finishing write tid 2 to smithi10548823-14 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:1987: finishing write tid 1 to smithi10548823-13 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:1985: finishing write tid 3 to smithi10548823-14 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:1985: finishing write tid 4 to smithi10548823-14 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:1985: finishing write tid 5 to smithi10548823-14 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:1985: oid 14 updating version 0 to 141 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:1985: oid 14 updating version 141 to 142 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:1985: oid 14 version 142 is already newer than 139 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:1985: oid 14 version 142 is already newer than 140 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 14 v 142 (ObjNum 411 snap 46 seq_num 411) dirty exists 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:1985: left oid 14 (ObjNum 411 snap 46 seq_num 411) 2024-04-20T02:53:20.204 INFO:tasks.rados.rados.0.smithi105.stdout:1981: done (10 left) 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1984: done (9 left) 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1985: done (8 left) 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1986: done (7 left) 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1988: done (6 left) 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1989: done (5 left) 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1990: done (4 left) 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1991: snap_create 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1987: finishing write tid 2 to smithi10548823-13 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1987: finishing write tid 3 to smithi10548823-13 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1987: finishing write tid 4 to smithi10548823-13 2024-04-20T02:53:20.205 INFO:tasks.rados.rados.0.smithi105.stdout:1987: finishing write tid 5 to smithi10548823-13 2024-04-20T02:53:20.206 INFO:tasks.rados.rados.0.smithi105.stdout:1987: oid 13 updating version 0 to 488 2024-04-20T02:53:20.206 INFO:tasks.rados.rados.0.smithi105.stdout:1987: oid 13 version 488 is already newer than 487 2024-04-20T02:53:20.206 INFO:tasks.rados.rados.0.smithi105.stdout:1987: oid 13 updating version 488 to 490 2024-04-20T02:53:20.206 INFO:tasks.rados.rados.0.smithi105.stdout:1987: oid 13 version 490 is already newer than 489 2024-04-20T02:53:20.206 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 13 v 490 (ObjNum 412 snap 46 seq_num 412) dirty exists 2024-04-20T02:53:20.206 INFO:tasks.rados.rados.0.smithi105.stdout:1987: left oid 13 (ObjNum 412 snap 46 seq_num 412) 2024-04-20T02:53:20.258 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:20.258+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:20.329 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:20.329 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:20.329 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:20.329 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:20.329 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.329 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:20.329 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.593068112, 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.330 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.331 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.332 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.333 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:53:20.334 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.335 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.561779051, 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.336 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.337 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.338 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.339 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.340 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:20.341 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.55708859500001, 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:20.342 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.343 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.344 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.345 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.346 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:20.347 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.550332126, 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:20.348 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.349 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.350 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.351 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.352 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:20.353 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.540516615, 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.354 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:53:20.355 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:53:20.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:53:20.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:53:20.356 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.356 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.356 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.356 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:53:20.356 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:20.357 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.358 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.359 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.53947268300001, 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.360 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:53:20.361 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.362 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:20.363 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.364 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.365 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.539451523, 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.366 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.367 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.368 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:53:20.369 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.370 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.331880942, 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.371 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.372 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.373 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.374 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.31768629699999, 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.375 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.376 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.377 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:53:20.378 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.31171904000001, 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:20.379 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.380 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.381 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.382 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.30331303700001, 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.383 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.384 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:53:20.385 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:53:20.386 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.387 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 158.19968099600001, 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.388 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.389 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.390 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:53:20.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:53:20.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.391 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:53:20.391 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:53:20.391 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.391 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.391 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 157.969084686, 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.392 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.393 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:53:20.394 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.395 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 157.942140537, 2024-04-20T02:53:20.396 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.397 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:53:20.398 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:53:20.399 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.400 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 157.91678227, 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.401 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.402 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.403 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.404 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 157.888641826, 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.405 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.406 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.407 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:53:20.408 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 157.85127322100001, 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:20.409 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.410 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.411 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.412 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 157.85097629800001, 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.413 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.414 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.415 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.416 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 157.84589317999999, 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.417 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:20.418 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:53:20.419 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:53:20.420 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 157.81889062100001, 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:53:20.421 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.422 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.423 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.424 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.425 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:20.427 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:20.427 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:20.427+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:20.527 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:20.527+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:20.556 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:20.556 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:20.556 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:20.566 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:20.629 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:20.629+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:20.729 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:20.729+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:20.734 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:20.734 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:20.735 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:20.735 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:20.735 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:20.745 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:20.831 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:20.831+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:20.915 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:20.915 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:20.915 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:20.915 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:20.915 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.915 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 181.310663568, 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:20.916 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.917 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.918 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.919 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:53:20.920 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:53:20.921 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 159.43349412800001, 2024-04-20T02:53:20.922 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.923 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.924 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.925 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:53:20.926 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.927 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.928 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 159.07370847600001, 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.929 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:53:20.930 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:53:20.931 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:53:20.932 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 159.063344871, 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.933 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:53:20.934 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.935 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:53:20.936 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.77370216700001, 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.937 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.938 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.939 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.940 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.941 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:20.942 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.76856505699999, 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.943 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.944 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.945 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.946 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.947 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.76829204500001, 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:53:20.948 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.949 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.950 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:53:20.951 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:20.952 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.953 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.76826266, 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:20.954 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.955 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:53:20.956 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:20.957 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:53:20.958 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 118.080935361, 2024-04-20T02:53:20.959 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:53:20.960 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.961 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.962 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:20.963 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 92.633455139000006, 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.964 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.965 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.966 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.967 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 92.616186705000004, 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.968 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.969 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:53:20.970 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.971 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 69.443040909999993, 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.972 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.973 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:53:20.974 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:53:20.975 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3881 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'232, mlcod=141'228)", 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.495918369999998, 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050172267999999999, 2024-04-20T02:53:20.976 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.977 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418659+0000", 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419496+0000", 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00083715799999999996 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419513+0000", 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7489000000000001e-05 2024-04-20T02:53:20.978 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419524+0000", 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0118e-05 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429881+0000", 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010357762 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429905+0000", 2024-04-20T02:53:20.979 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3546000000000001e-05 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468790+0000", 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038884821 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468832+0000", 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2341e-05 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.980 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3898 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'133, mlcod=133'129)", 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.443759397999997, 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038834664999999997, 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.981 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470817+0000", 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471051+0000", 2024-04-20T02:53:20.982 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023384600000000001 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471066+0000", 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4205e-05 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471074+0000", 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4090000000000003e-06 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.983 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472816+0000", 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017419429999999999 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472843+0000", 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7296000000000001e-05 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509592+0000", 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036748496999999998 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.984 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509653+0000", 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1707999999999999e-05 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3899 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'134, mlcod=133'129)", 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.426492961999998, 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038218993999999999, 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.985 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488084+0000", 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.986 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488262+0000", 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017768600000000001 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488274+0000", 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2388000000000001e-05 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488283+0000", 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3840000000000003e-06 2024-04-20T02:53:20.987 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489094+0000", 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081045100000000001 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489119+0000", 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4751000000000001e-05 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526264+0000", 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037144912000000002 2024-04-20T02:53:20.988 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526304+0000", 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0513000000000003e-05 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3891 4.7 e146/145 4:e2c0d65b:::smithi10548823-46:head v 146'105, mlcod=143'100)", 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.425512867000002, 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041132308999999999, 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.989 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:20.990 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489072+0000", 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9539999999999998e-06 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489082+0000", 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0329000000000001e-05 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.991 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489089+0000", 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4959999999999999e-06 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489814+0000", 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072423699999999997 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489837+0000", 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3218999999999999e-05 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:20.992 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530157+0000", 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040319762000000002 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530198+0000", 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0886000000000001e-05 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.7 146/145 [PushOp(4:e2c0d65b:::smithi10548823-46:43, version: 143'96, data_included: [426944~820638,1468778~227938], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:e2c0d65b:::smithi10548823-46:43@143'96, size: 3311012, copy_subset: [0~3311012], clone_subset: {}, snapset: 45=[]:{3f=[3f,39],43=[41],45=[45]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1696716, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.424301515000003, 2024-04-20T02:53:20.993 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039876652999999998, 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.994 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490276+0000", 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490785+0000", 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050941300000000001 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490804+0000", 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9080999999999999e-05 2024-04-20T02:53:20.995 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490823+0000", 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.855e-05 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498138+0000", 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007315004 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498153+0000", 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5515e-05 2024-04-20T02:53:20.996 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530153+0000", 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031999845999999998 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4016 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'146, mlcod=147'143)", 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.651997268999999, 2024-04-20T02:53:20.997 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054600045999999999, 2024-04-20T02:53:20.998 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:20.998 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:20.998 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:20.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:20.998 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:20.998 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.998 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.998 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.998 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262580+0000", 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262733+0000", 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015289700000000001 2024-04-20T02:53:20.999 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262746+0000", 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.375e-05 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262753+0000", 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6159999999999998e-06 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262817+0000", 2024-04-20T02:53:21.000 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4202999999999995e-05 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262842+0000", 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5296e-05 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317137+0000", 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054294664999999999 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317181+0000", 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:21.001 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4017 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'147, mlcod=147'143)", 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.636950583000001, 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039585437000000001, 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:21.002 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277626+0000", 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278058+0000", 2024-04-20T02:53:21.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043228299999999999 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278074+0000", 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5775000000000001e-05 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278083+0000", 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2620000000000004e-06 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.004 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286849+0000", 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087652210000000001 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286881+0000", 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1919999999999999e-05 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317188+0000", 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030307392999999998 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317213+0000", 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4953999999999999e-05 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4340 4.8 e163/114 4:1a8c7fb2:::smithi10548823-47:head v 163'162, mlcod=159'158)", 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 18.002274197999999, 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.04004311, 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912303+0000", 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912722+0000", 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00041911999999999999 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912738+0000", 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6294e-05 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912748+0000", 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6059999999999992e-06 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912840+0000", 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2143000000000002e-05 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912867+0000", 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7121e-05 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952298+0000", 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039430999000000001 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952347+0000", 2024-04-20T02:53:21.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8990999999999999e-05 2024-04-20T02:53:21.010 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.010 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.010 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.010 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.010 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.010 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:21.012 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:21.012 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:21.012+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:21.114 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:21.113+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:21.141 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:21.141 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:21.141 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:21.141 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:21.146 INFO:tasks.rados.rados.0.smithi105.stdout:1987: done (4 left) 2024-04-20T02:53:21.146 INFO:tasks.rados.rados.0.smithi105.stdout:1991: done (3 left) 2024-04-20T02:53:21.146 INFO:tasks.rados.rados.0.smithi105.stdout:1992: read oid 24 snap -1 2024-04-20T02:53:21.146 INFO:tasks.rados.rados.0.smithi105.stdout:1992: expect deleted 2024-04-20T02:53:21.146 INFO:tasks.rados.rados.0.smithi105.stdout:1993: read oid 19 snap -1 2024-04-20T02:53:21.146 INFO:tasks.rados.rados.0.smithi105.stdout:1993: expect (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:53:21.146 INFO:tasks.rados.rados.0.smithi105.stdout:1994: snap_remove snap 46 2024-04-20T02:53:21.154 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:21.166 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:53:21.215 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:21.215+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:21.315 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:21.316+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:21.325 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:21.325 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:21.325 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:21.325 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:21.325 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:21.335 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:21.417 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:21.416+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:21.466 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:21.466 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:21.466 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:21.466 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:21.466 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.466 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 132.01658402800001, 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.467 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.468 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:53:21.469 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:53:21.470 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.966401261, 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.471 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.472 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.473 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.474 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:21.475 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.966335812, 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.476 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:53:21.477 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:53:21.478 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:53:21.479 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.966310314, 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.480 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.481 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:53:21.482 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:53:21.483 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.484 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2524 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1723287~549075] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 93.178800831000004, 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054106970999999997, 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2524 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:21.485 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286419+0000", 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.486 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286924+0000", 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050494600000000004 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286944+0000", 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9833000000000001e-05 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286957+0000", 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2901000000000001e-05 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.487 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298074+0000", 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011117320999999999 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298151+0000", 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6909999999999994e-05 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298274+0000", 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012279700000000001 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.488 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326267+0000", 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027993284 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331226+0000", 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049590559999999999 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.489 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340442+0000", 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092155740000000007 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340488+0000", 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6208000000000002e-05 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340528+0000", 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0277000000000001e-05 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.490 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3637 4.1 e137/60 4:80c7a726:::smithi10548823-19:head v 137'124, mlcod=120'120)", 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.840813271999998, 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052640605, 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.491 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624408+0000", 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624537+0000", 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012880199999999999 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.492 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624548+0000", 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0216e-05 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624556+0000", 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6179999999999995e-06 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624603+0000", 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6601999999999999e-05 2024-04-20T02:53:21.493 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624632+0000", 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9277000000000001e-05 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.676999+0000", 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052366639999999999 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.677049+0000", 2024-04-20T02:53:21.494 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0822999999999998e-05 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3880 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'231, mlcod=141'228)", 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.058318292000003, 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.061291667000000001, 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.495 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406903+0000", 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.496 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407359+0000", 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00045623700000000002 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407370+0000", 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1052e-05 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407377+0000", 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1080000000000004e-06 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.497 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407429+0000", 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2033999999999997e-05 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407458+0000", 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8050000000000001e-05 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468161+0000", 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060703628000000003 2024-04-20T02:53:21.498 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468196+0000", 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4344999999999997e-05 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3888 4.7 4:e2c0d65b:::smithi10548823-46:head [write 413538~536345,stat] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.039765164000002, 2024-04-20T02:53:21.499 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.059175945000000001, 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3888 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.500 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425456+0000", 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426097+0000", 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00064115099999999996 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.501 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426113+0000", 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6118000000000001e-05 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426122+0000", 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.115e-06 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426179+0000", 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6815999999999999e-05 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.502 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426316+0000", 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000136606 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426499+0000", 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000183191 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443652+0000", 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017153471999999999 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.503 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484036+0000", 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040384097000000001 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484231+0000", 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000194811 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484363+0000", 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013146300000000001 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.504 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484426+0000", 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3563000000000001e-05 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484633+0000", 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020662299999999999 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3889 4.7 4:e2c0d65b:::smithi10548823-46:head [write 1395221~583781] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:21.505 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.032871188999998, 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.057918795000000002, 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3889 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.506 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432350+0000", 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.507 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433490+0000", 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001140606 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433506+0000", 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5828000000000002e-05 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433515+0000", 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8279999999999992e-06 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.508 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443692+0000", 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010177222 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443839+0000", 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014662 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.444072+0000", 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023362200000000001 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.509 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484114+0000", 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040041641000000003 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484244+0000", 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012987200000000001 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.510 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484279+0000", 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5301e-05 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490055+0000", 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057763459999999999 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490115+0000", 2024-04-20T02:53:21.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9320999999999997e-05 2024-04-20T02:53:21.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490270+0000", 2024-04-20T02:53:21.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015498199999999999 2024-04-20T02:53:21.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.512 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.512 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3890 4.7 4:e2c0d65b:::smithi10548823-46:head [write 2556103~658617] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.024857087999997, 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066878325000000002, 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3890 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.513 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440363+0000", 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441831+0000", 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001467632 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441842+0000", 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0540000000000001e-05 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441850+0000", 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7260000000000008e-06 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463536+0000", 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02168544 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463681+0000", 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000145086 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:21.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463941+0000", 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00025968600000000002 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484317+0000", 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020376452 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490318+0000", 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060012490000000002 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.517 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490398+0000", 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9598000000000006e-05 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506988+0000", 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016589706999999999 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507066+0000", 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8299000000000004e-05 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507243+0000", 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000177503 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3891 4.7 4:e2c0d65b:::smithi10548823-46:head [setxattr _header (57) in=7b,truncate 3613538] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.023361579000003, 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088590031, 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.519 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3891 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441859+0000", 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441864+0000", 2024-04-20T02:53:21.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8069999999999999e-06 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441869+0000", 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0989999999999998e-06 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441875+0000", 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5820000000000001e-06 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484675+0000", 2024-04-20T02:53:21.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042799867999999998 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484817+0000", 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014236899999999999 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485048+0000", 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023057199999999999 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490359+0000", 2024-04-20T02:53:21.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053116819999999999 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506963+0000", 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016603562999999998 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507256+0000", 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00029288699999999998 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530316+0000", 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023060343000000001 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530356+0000", 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9301999999999997e-05 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530451+0000", 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5037999999999994e-05 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3892 4.7 4:e2c0d65b:::smithi10548823-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.023341891000001, 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088912482000000001, 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3892 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441883+0000", 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9579999999999999e-06 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441889+0000", 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7499999999999997e-06 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441895+0000", 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485431+0000", 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043536321000000003 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485537+0000", 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010612599999999999 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530489+0000", 2024-04-20T02:53:21.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044951996000000001 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530598+0000", 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108937 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530793+0000", 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019441 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3898 4.8 4:1c68e542:::smithi10548823-8:head [write 447152~403196] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.998998808000003, 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.056917473000000003, 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3898 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466222+0000", 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466722+0000", 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00049941299999999998 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466734+0000", 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2860999999999999e-05 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466743+0000", 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7159999999999992e-06 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466809+0000", 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6165000000000001e-05 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466928+0000", 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011869199999999999 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.467158+0000", 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000230251 2024-04-20T02:53:21.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498338+0000", 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.03117996 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.522958+0000", 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024619318000000001 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523002+0000", 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4675999999999997e-05 2024-04-20T02:53:21.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523082+0000", 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9803999999999997e-05 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523141+0000", 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8767000000000002e-05 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.537 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3899 4.8 4:1c68e542:::smithi10548823-8:head [write 1512185~587450] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.991071040999998, 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060518556000000001, 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3899 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.538 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474149+0000", 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474686+0000", 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053696199999999999 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474703+0000", 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.615e-05 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474713+0000", 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0829000000000001e-05 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478533+0000", 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003819687 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478640+0000", 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106974 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478766+0000", 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125859 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523803+0000", 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045036616000000002 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526432+0000", 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026291489999999999 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534589+0000", 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081572209999999992 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534623+0000", 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534670+0000", 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6968999999999997e-05 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4016 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 665643~697786] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:21.543 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.203902276000001, 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060676251, 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4016 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261318+0000", 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262081+0000", 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076285299999999997 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262095+0000", 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3986e-05 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262118+0000", 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2407000000000001e-05 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262164+0000", 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6248999999999997e-05 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262238+0000", 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4202000000000005e-05 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262363+0000", 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012437799999999999 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314944+0000", 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052581687000000002 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317407+0000", 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024628300000000001 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321887+0000", 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0044803020000000002 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321941+0000", 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3769e-05 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321996+0000", 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4988000000000003e-05 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4017 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 2039745~753122] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.196272477999997, 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053146383999999998, 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.549 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4017 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:21.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268948+0000", 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269678+0000", 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072975799999999995 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269693+0000", 2024-04-20T02:53:21.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.453e-05 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269704+0000", 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1623e-05 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277190+0000", 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074859849999999997 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277276+0000", 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5809999999999993e-05 2024-04-20T02:53:21.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277405+0000", 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012914600000000001 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314952+0000", 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037546931999999998 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317441+0000", 2024-04-20T02:53:21.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024889840000000001 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322031+0000", 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004589465 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322059+0000", 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322096+0000", 2024-04-20T02:53:21.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591999999999998e-05 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4074 4.11 4:8b91c759:::smithi10548823-5:head [write 1591248~408851] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 29.323305089000002, 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053876146, 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.555 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4074 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141916+0000", 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142740+0000", 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000823616 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142750+0000", 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6150000000000003e-06 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142756+0000", 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9419999999999998e-06 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2061999999999997e-05 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142815+0000", 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5050000000000004e-06 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174069+0000", 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031253933999999997 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174193+0000", 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012386 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174438+0000", 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00024561400000000001 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187472+0000", 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013033375999999999 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.188278+0000", 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00080595500000000002 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195551+0000", 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072733340000000002 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195631+0000", 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9882999999999994e-05 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195793+0000", 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000162223 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4075 4.11 4:8b91c759:::smithi10548823-5:head [setxattr _header (56) in=7b,truncate 2000099] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:21.561 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 29.322454156999999, 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084064655000000002, 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4075 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:21.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142767+0000", 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142778+0000", 2024-04-20T02:53:21.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0709e-05 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142783+0000", 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4539999999999998e-06 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142788+0000", 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2850000000000002e-06 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142845+0000", 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6919000000000002e-05 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142856+0000", 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0936e-05 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186635+0000", 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043778897999999997 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186756+0000", 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121087 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186967+0000", 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021064599999999999 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.189477+0000", 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025104929999999999 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:21.566 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.190395+0000", 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00091717399999999998 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226595+0000", 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036200195999999997 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226661+0000", 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6465999999999997e-05 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226833+0000", 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000171471 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4076 4.11 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 29.322413150999999, 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084363853000000003, 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:21.568 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4076 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142808+0000", 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142811+0000", 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9229999999999998e-06 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142818+0000", 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4420000000000003e-06 2024-04-20T02:53:21.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142823+0000", 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5199999999999997e-06 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142875+0000", 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1759999999999999e-05 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142880+0000", 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6639999999999997e-06 2024-04-20T02:53:21.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187624+0000", 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044743949999999998 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187726+0000", 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0001022 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226882+0000", 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039155862 2024-04-20T02:53:21.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227003+0000", 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120715 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227173+0000", 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017032899999999999 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.573 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4183 3.2 e154/49 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head v 154'234, mlcod=154'232)", 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 27.812284217999999, 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.063121930000000007, 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:21.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652936+0000", 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657053+0000", 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041172520000000001 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:21.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657067+0000", 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3865e-05 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657076+0000", 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5390000000000007e-06 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657121+0000", 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5114000000000002e-05 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:21.576 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657138+0000", 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7600000000000001e-05 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716021+0000", 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.058882663000000002 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716060+0000", 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8826999999999999e-05 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:21.577 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:21.578 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:21.579 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:21.580 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:21.579+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:21.680 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:21.679+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:21.707 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:21.707 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:21.708 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:21.708 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:21.717 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:21.781 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:21.781+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:21.846 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:21.846 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:21.846 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:21.846 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:21.846 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:21.856 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:21.882 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:21.882+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:21.983 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:21.983+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:21.983 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:21.983 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:21.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:21.984 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-04-20T02:53:21.984 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:21.994 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:22.077 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:53:22.083 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:22.083+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:22.135 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:22.135 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:22.135 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:22.135 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:22.146 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:22.147 INFO:tasks.rados.rados.0.smithi105.stdout:1994: done (5 left) 2024-04-20T02:53:22.147 INFO:tasks.rados.rados.0.smithi105.stdout:1995: write (excl) oid 20 current snap is 48 2024-04-20T02:53:22.147 INFO:tasks.rados.rados.0.smithi105.stdout:1995: seq_num 413 ranges {541291=560899,1884475=521471} 2024-04-20T02:53:22.150 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:53:22.154 INFO:tasks.rados.rados.0.smithi105.stdout:1995: writing smithi10548823-20 from 541291 to 1102190 tid 1 2024-04-20T02:53:22.162 INFO:tasks.rados.rados.0.smithi105.stdout:1995: writing smithi10548823-20 from 1884475 to 2405946 tid 2 2024-04-20T02:53:22.162 INFO:tasks.rados.rados.0.smithi105.stdout:1996: rollback oid 16 current snap is 48 2024-04-20T02:53:22.162 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 16 to 47 2024-04-20T02:53:22.162 INFO:tasks.rados.rados.0.smithi105.stdout:1997: read oid 43 snap -1 2024-04-20T02:53:22.162 INFO:tasks.rados.rados.0.smithi105.stdout:1997: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:53:22.163 INFO:tasks.rados.rados.0.smithi105.stdout:1998: read oid 15 snap 47 2024-04-20T02:53:22.163 INFO:tasks.rados.rados.0.smithi105.stdout:1998: expect (ObjNum 400 snap 46 seq_num 400) 2024-04-20T02:53:22.163 INFO:tasks.rados.rados.0.smithi105.stdout:1999: tier_promote oid 22 2024-04-20T02:53:22.163 INFO:tasks.rados.rados.0.smithi105.stdout:2000: set_chunk oid 35 target oid 2024-04-20T02:53:22.179 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 3, 6, 7, 5, 2, 4] 2024-04-20T02:53:22.179 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:53:22.179 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-20T02:53:22.180 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-20T02:53:22.218 INFO:tasks.rados.rados.0.smithi105.stdout:2000: set_chunk oid 35 offset: 204288 length: 1084416 target oid d9a5132369b3468a8e29053bedc453d2c2e95fbd7cc4575c21c1cdaa30e50108 offset: 0 snap -1 2024-04-20T02:53:22.218 INFO:tasks.rados.rados.0.smithi105.stdout:2001: snap_remove snap 45 2024-04-20T02:53:22.218 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:22.218+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:22.244 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:22.265 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:22.284 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:22.285 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:22.284+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:22.303 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:22.316 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:22.316 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:22.316 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:22.316 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:22.316 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:22.324 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:22.326 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:22.343 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:22.363 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:22.383 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:22.384 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:22.384 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:22.384 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:22.384 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:22.384 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:22.384 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:22.384 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:22.384 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:22.385 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:22.385+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:22.486 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:22.486+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:22.494 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:22.494 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:22.505 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:22.521 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:53:22.521 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:22.521 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 183, 2024-04-20T02:53:22.521 INFO:teuthology.orchestra.run.smithi105.stdout: "fsid": "a524a365-eaec-4ebd-aa4d-0d76d55a252f", 2024-04-20T02:53:22.521 INFO:teuthology.orchestra.run.smithi105.stdout: "created": "2024-04-20T02:50:07.356902+0000", 2024-04-20T02:53:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "modified": "2024-04-20T02:53:22.143060+0000", 2024-04-20T02:53:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "last_up_change": "2024-04-20T02:53:06.173366+0000", 2024-04-20T02:53:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "last_in_change": "2024-04-20T02:52:50.391054+0000", 2024-04-20T02:53:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T02:53:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_num": 5801984, 2024-04-20T02:53:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_set": [ 2024-04-20T02:53:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "noscrub", 2024-04-20T02:53:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "pglog_hardlimit", 2024-04-20T02:53:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "purged_snapdirs", 2024-04-20T02:53:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_deletes", 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "sortbitwise" 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_version": 7, 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_snapshot": "", 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_max": 4, 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "max_osd": 8, 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "require_min_compat_client": "luminous", 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "min_compat_client": "luminous", 2024-04-20T02:53:22.523 INFO:teuthology.orchestra.run.smithi105.stdout: "require_osd_release": "reef", 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_crimson": false, 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "pools": [ 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 1, 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": ".mgr", 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:17.498569+0000", 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool", 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:53:22.524 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 1, 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 1, 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 1, 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 1, 2024-04-20T02:53:22.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 1, 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "0.0", 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 0, 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 0, 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 0, 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "0'0", 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "0'0" 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "17", 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:53:22.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 0, 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 0, 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:53:22.527 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:53:22.528 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:53:22.529 INFO:teuthology.orchestra.run.smithi105.stdout: "options": { 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_max": 32, 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_min": 1 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": { 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr": {} 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 7.8899998664855957, 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 7.8899998664855957, 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 0.37999999523162842, 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 3, 2024-04-20T02:53:22.530 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 3, 2024-04-20T02:53:22.531 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:53:22.531 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:53:22.531 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:53:22.531 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.531 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 2, 2024-04-20T02:53:22.531 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": "rbd", 2024-04-20T02:53:22.531 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:25.955448+0000", 2024-04-20T02:53:22.531 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool", 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:53:22.532 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 8, 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 8, 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 8, 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 8, 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 8, 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "0.0", 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 0, 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 0, 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 0, 2024-04-20T02:53:22.533 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "0'0", 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "0'0" 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "20", 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 0, 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 0, 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:53:22.534 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:53:22.535 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:53:22.536 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "options": {}, 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": { 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd": {} 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 2, 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 2, 2024-04-20T02:53:22.537 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 0.75, 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 1.5, 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 1.5, 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 3, 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": "low_tier", 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:33.168857+0000", 2024-04-20T02:53:22.538 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool", 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 4, 2024-04-20T02:53:22.539 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 4, 2024-04-20T02:53:22.540 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 4, 2024-04-20T02:53:22.540 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 4, 2024-04-20T02:53:22.540 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 4, 2024-04-20T02:53:22.540 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:53:22.540 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "0.0", 2024-04-20T02:53:22.540 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 0, 2024-04-20T02:53:22.540 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 0, 2024-04-20T02:53:22.540 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 0, 2024-04-20T02:53:22.540 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "0'0", 2024-04-20T02:53:22.540 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "0'0" 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "23", 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 0, 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 0, 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:53:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:53:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:53:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:53:22.544 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:53:22.544 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:53:22.544 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:53:22.544 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:53:22.544 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:53:22.544 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:53:22.544 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:53:22.544 INFO:teuthology.orchestra.run.smithi105.stdout: "options": {}, 2024-04-20T02:53:22.544 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": {}, 2024-04-20T02:53:22.544 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 3.9700000286102295, 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 3.9700000286102295, 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 0.62999999523162842, 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 2.5, 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 2.5, 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.546 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 4, 2024-04-20T02:53:22.546 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": "unique_pool_0", 2024-04-20T02:53:22.546 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:33.714259+0000", 2024-04-20T02:53:22.546 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 8193, 2024-04-20T02:53:22.546 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-20T02:53:22.546 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:53:22.546 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:53:22.546 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:53:22.546 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:53:22.547 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:53:22.547 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:53:22.547 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:53:22.547 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:53:22.547 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:53:22.547 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:53:22.547 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 19, 2024-04-20T02:53:22.547 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 16, 2024-04-20T02:53:22.548 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 16, 2024-04-20T02:53:22.548 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 16, 2024-04-20T02:53:22.548 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 18, 2024-04-20T02:53:22.548 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:53:22.548 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "4.13", 2024-04-20T02:53:22.548 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 181, 2024-04-20T02:53:22.548 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 180, 2024-04-20T02:53:22.548 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 180, 2024-04-20T02:53:22.548 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "176'294", 2024-04-20T02:53:22.548 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "174'249" 2024-04-20T02:53:22.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.549 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "183", 2024-04-20T02:53:22.549 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:53:22.549 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "183", 2024-04-20T02:53:22.549 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "183", 2024-04-20T02:53:22.549 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:53:22.549 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:53:22.549 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 95, 2024-04-20T02:53:22.549 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 183, 2024-04-20T02:53:22.549 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:53:22.550 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:53:22.551 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:53:22.552 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:53:22.552 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:53:22.552 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:53:22.552 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:53:22.552 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:53:22.552 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:53:22.553 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:53:22.553 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:53:22.553 INFO:teuthology.orchestra.run.smithi105.stdout: "options": { 2024-04-20T02:53:22.553 INFO:teuthology.orchestra.run.smithi105.stdout: "dedup_cdc_chunk_size": 131072, 2024-04-20T02:53:22.553 INFO:teuthology.orchestra.run.smithi105.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-04-20T02:53:22.553 INFO:teuthology.orchestra.run.smithi105.stdout: "dedup_tier": 3, 2024-04-20T02:53:22.553 INFO:teuthology.orchestra.run.smithi105.stdout: "fingerprint_algorithm": "sha256" 2024-04-20T02:53:22.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.553 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": { 2024-04-20T02:53:22.553 INFO:teuthology.orchestra.run.smithi105.stdout: "rados": {} 2024-04-20T02:53:22.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.554 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:53:22.554 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 2.5199999809265137, 2024-04-20T02:53:22.554 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 2.5199999809265137, 2024-04-20T02:53:22.554 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 0.75, 2024-04-20T02:53:22.554 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 1.8899999856948853, 2024-04-20T02:53:22.554 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 1.8899999856948853, 2024-04-20T02:53:22.554 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:53:22.554 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:53:22.554 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: "osds": [ 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 0, 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6", 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 0.8316650390625, 2024-04-20T02:53:22.555 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:22.556 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:53:22.556 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:53:22.556 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 15, 2024-04-20T02:53:22.556 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 181, 2024-04-20T02:53:22.556 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:53:22.556 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:22.556 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:22.556 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6812", 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.557 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6813", 2024-04-20T02:53:22.558 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:53:22.558 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.558 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.558 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:22.558 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.558 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.558 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6815", 2024-04-20T02:53:22.558 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6814", 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.559 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.560 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6812/43330", 2024-04-20T02:53:22.560 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6813/43330", 2024-04-20T02:53:22.560 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6815/43330", 2024-04-20T02:53:22.560 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6814/43330", 2024-04-20T02:53:22.560 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:22.560 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:22.560 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:22.560 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 1, 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "c5ae8a64-fc7c-47ab-af6a-4b762a840645", 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 0.8221893310546875, 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 13, 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 177, 2024-04-20T02:53:22.561 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:53:22.562 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:22.562 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:22.562 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.562 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.562 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6804", 2024-04-20T02:53:22.562 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:53:22.562 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6805", 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6807", 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6806", 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6804/43331", 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6805/43331", 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6807/43331", 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6806/43331", 2024-04-20T02:53:22.565 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:22.566 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:22.566 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:22.566 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.566 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 2, 2024-04-20T02:53:22.566 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "336e852e-5fa6-4ac0-b053-d31ad40d7c4e", 2024-04-20T02:53:22.566 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:22.566 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 0, 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 0, 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 13, 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 121, 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 138, 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 27, 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 122, 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.568 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.568 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6800", 2024-04-20T02:53:22.568 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66776 2024-04-20T02:53:22.568 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.568 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.568 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:22.568 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6801", 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66776 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6803", 2024-04-20T02:53:22.569 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66776 2024-04-20T02:53:22.570 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.570 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.570 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:22.570 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.570 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.570 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6802", 2024-04-20T02:53:22.571 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66776 2024-04-20T02:53:22.571 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.571 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.571 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6800/66776", 2024-04-20T02:53:22.571 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6801/66776", 2024-04-20T02:53:22.571 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6803/66776", 2024-04-20T02:53:22.571 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6802/66776", 2024-04-20T02:53:22.571 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:22.571 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 3, 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "695cab3a-74d8-4046-ac61-2656dc0d893e", 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:53:22.572 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 13, 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 176, 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6808", 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.574 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:22.574 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.574 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.574 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6809", 2024-04-20T02:53:22.574 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:22.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.574 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6811", 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:22.575 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6810", 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6808/43329", 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6809/43329", 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6811/43329", 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6810/43329", 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:22.576 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:22.577 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:22.577 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.577 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 4, 2024-04-20T02:53:22.577 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "1d41da44-d894-40ac-acaa-aa0df15296c6", 2024-04-20T02:53:22.577 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:22.577 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 0, 2024-04-20T02:53:22.577 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 0, 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 0.9784088134765625, 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 15, 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 149, 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 167, 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 145, 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 150, 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.578 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6808", 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66272 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6809", 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66272 2024-04-20T02:53:22.579 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.580 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.580 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:22.580 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.580 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.580 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6811", 2024-04-20T02:53:22.580 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66272 2024-04-20T02:53:22.580 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6810", 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66272 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.582 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6808/66272", 2024-04-20T02:53:22.582 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6809/66272", 2024-04-20T02:53:22.582 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6811/66272", 2024-04-20T02:53:22.582 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6810/66272", 2024-04-20T02:53:22.582 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:22.582 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:22.582 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:22.582 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.583 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 5, 2024-04-20T02:53:22.583 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "3e1c33fa-6c8c-4898-99bc-462ecef25f30", 2024-04-20T02:53:22.583 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:22.583 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:22.583 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:53:22.583 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:22.583 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 15, 2024-04-20T02:53:22.583 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 25, 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 38, 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 171, 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 26, 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6802", 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6804", 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.585 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6807", 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6806", 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.587 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6802/52590", 2024-04-20T02:53:22.587 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6804/52590", 2024-04-20T02:53:22.587 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6807/52590", 2024-04-20T02:53:22.587 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6806/52590", 2024-04-20T02:53:22.587 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:22.587 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:22.587 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:22.587 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 6, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "fd1fc124-7cf8-4628-9076-1f27d6385676", 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 13, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 178, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:22.588 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.589 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.589 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6800", 2024-04-20T02:53:22.589 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:22.589 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.589 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.589 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:22.589 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6801", 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6805", 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.590 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.591 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:22.591 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.591 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.591 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6803", 2024-04-20T02:53:22.591 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:22.591 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.591 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6800/43329", 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6801/43329", 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6805/43329", 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6803/43329", 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 7, 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "53ae8193-5c3e-4698-aacd-df2ad1db9143", 2024-04-20T02:53:22.592 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 15, 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 154, 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.593 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.594 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6812", 2024-04-20T02:53:22.594 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:53:22.594 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.594 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.594 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:22.594 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.594 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.594 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.594 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6813", 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6815", 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.595 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6814", 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6812/43351", 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6813/43351", 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6815/43351", 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6814/43351", 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:22.596 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_xinfo": [ 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 0, 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:44.296659+0000", 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:53:22.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 1, 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:23.214623+0000", 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 2, 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:22.598 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 65536, 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:56.806719+0000", 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 122 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 3, 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:58.973108+0000", 2024-04-20T02:53:22.599 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 4, 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 49625, 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:38.383284+0000", 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 150 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 5, 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:22.600 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:29.483129+0000", 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 26 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 6, 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:22.601 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:18.346058+0000", 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 7, 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:29.823916+0000", 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:22.602 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_upmap": [ 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: "pgid": "2.0", 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: "osds": [ 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: 6, 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: 0, 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: 2 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_upmap_items": [], 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_upmap_primaries": [], 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_temp": [], 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_temp": [], 2024-04-20T02:53:22.603 INFO:teuthology.orchestra.run.smithi105.stdout: "blocklist": {}, 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "range_blocklist": {}, 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profiles": { 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "default": { 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "crush-failure-domain": "osd", 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "k": "2", 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "m": "1", 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "plugin": "jerasure", 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "technique": "reed_sol_van" 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps_queue": [ 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 4, 2024-04-20T02:53:22.604 INFO:teuthology.orchestra.run.smithi105.stdout: "snaps": [ 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 57, 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 63, 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 65, 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 69, 2024-04-20T02:53:22.605 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 71, 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 75, 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 77, 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 2 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.606 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 80, 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 83, 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 3 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 90, 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 93, 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.607 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "new_removed_snaps": [ 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 4, 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "snaps": [ 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 93, 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "new_purged_snaps": [], 2024-04-20T02:53:22.608 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_node_flags": {}, 2024-04-20T02:53:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "device_class_flags": {}, 2024-04-20T02:53:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_mode": { 2024-04-20T02:53:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_mode_enabled": false, 2024-04-20T02:53:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_bucket_count": 0, 2024-04-20T02:53:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "degraded_stretch_mode": 0, 2024-04-20T02:53:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "recovering_stretch_mode": 0, 2024-04-20T02:53:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_mode_bucket": 0 2024-04-20T02:53:22.609 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:22.609 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:22.611 INFO:tasks.thrashosds.thrasher:j is {'epoch': 183, 'fsid': 'a524a365-eaec-4ebd-aa4d-0d76d55a252f', 'created': '2024-04-20T02:50:07.356902+0000', 'modified': '2024-04-20T02:53:22.143060+0000', 'last_up_change': '2024-04-20T02:53:06.173366+0000', 'last_in_change': '2024-04-20T02:52:50.391054+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T02:50:17.498569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-20T02:50:25.955448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-04-20T02:50:33.168857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_acting': 3.9700000286102295, 'score_stable': 3.9700000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-20T02:50:33.714259+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 19, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 18, 'last_pg_merge_meta': {'source_pgid': '4.13', 'ready_epoch': 181, 'last_epoch_started': 180, 'last_epoch_clean': 180, 'source_version': "176'294", 'target_version': "174'249"}, 'last_change': '183', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '183', 'last_force_op_resend_preluminous': '183', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 95, 'snap_epoch': 183, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_acting': 2.5199999809265137, 'score_stable': 2.5199999809265137, 'optimal_score': 0.75, 'raw_score_acting': 1.8899999856948853, 'raw_score_stable': 1.8899999856948853, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6', 'up': 1, 'in': 1, 'weight': 0.8316650390625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 181, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6812', 'nonce': 43330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6813', 'nonce': 43330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6815', 'nonce': 43330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6814', 'nonce': 43330}]}, 'public_addr': '172.21.15.105:6812/43330', 'cluster_addr': '172.21.15.105:6813/43330', 'heartbeat_back_addr': '172.21.15.105:6815/43330', 'heartbeat_front_addr': '172.21.15.105:6814/43330', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5ae8a64-fc7c-47ab-af6a-4b762a840645', 'up': 1, 'in': 1, 'weight': 0.8221893310546875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 177, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6804', 'nonce': 43331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6805', 'nonce': 43331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6807', 'nonce': 43331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6806', 'nonce': 43331}]}, 'public_addr': '172.21.15.105:6804/43331', 'cluster_addr': '172.21.15.105:6805/43331', 'heartbeat_back_addr': '172.21.15.105:6807/43331', 'heartbeat_front_addr': '172.21.15.105:6806/43331', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '336e852e-5fa6-4ac0-b053-d31ad40d7c4e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 121, 'up_from': 138, 'up_thru': 27, 'down_at': 122, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6800', 'nonce': 66776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6801', 'nonce': 66776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6803', 'nonce': 66776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6802', 'nonce': 66776}]}, 'public_addr': '172.21.15.105:6800/66776', 'cluster_addr': '172.21.15.105:6801/66776', 'heartbeat_back_addr': '172.21.15.105:6803/66776', 'heartbeat_front_addr': '172.21.15.105:6802/66776', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '695cab3a-74d8-4046-ac61-2656dc0d893e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 176, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6808', 'nonce': 43329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6809', 'nonce': 43329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6811', 'nonce': 43329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6810', 'nonce': 43329}]}, 'public_addr': '172.21.15.105:6808/43329', 'cluster_addr': '172.21.15.105:6809/43329', 'heartbeat_back_addr': '172.21.15.105:6811/43329', 'heartbeat_front_addr': '172.21.15.105:6810/43329', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1d41da44-d894-40ac-acaa-aa0df15296c6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.9784088134765625, 'last_clean_begin': 15, 'last_clean_end': 149, 'up_from': 167, 'up_thru': 145, 'down_at': 150, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6808', 'nonce': 66272}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6809', 'nonce': 66272}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6811', 'nonce': 66272}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6810', 'nonce': 66272}]}, 'public_addr': '172.21.15.138:6808/66272', 'cluster_addr': '172.21.15.138:6809/66272', 'heartbeat_back_addr': '172.21.15.138:6811/66272', 'heartbeat_front_addr': '172.21.15.138:6810/66272', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '3e1c33fa-6c8c-4898-99bc-462ecef25f30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 15, 'last_clean_end': 25, 'up_from': 38, 'up_thru': 171, 'down_at': 26, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6802', 'nonce': 52590}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6804', 'nonce': 52590}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6807', 'nonce': 52590}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6806', 'nonce': 52590}]}, 'public_addr': '172.21.15.138:6802/52590', 'cluster_addr': '172.21.15.138:6804/52590', 'heartbeat_back_addr': '172.21.15.138:6807/52590', 'heartbeat_front_addr': '172.21.15.138:6806/52590', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fd1fc124-7cf8-4628-9076-1f27d6385676', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 178, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6800', 'nonce': 43329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6801', 'nonce': 43329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6805', 'nonce': 43329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6803', 'nonce': 43329}]}, 'public_addr': '172.21.15.138:6800/43329', 'cluster_addr': '172.21.15.138:6801/43329', 'heartbeat_back_addr': '172.21.15.138:6805/43329', 'heartbeat_front_addr': '172.21.15.138:6803/43329', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '53ae8193-5c3e-4698-aacd-df2ad1db9143', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6812', 'nonce': 43351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6813', 'nonce': 43351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6815', 'nonce': 43351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6814', 'nonce': 43351}]}, 'public_addr': '172.21.15.138:6812/43351', 'cluster_addr': '172.21.15.138:6813/43351', 'heartbeat_back_addr': '172.21.15.138:6815/43351', 'heartbeat_front_addr': '172.21.15.138:6814/43351', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:44.296659+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:23.214623+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-20T02:52:56.806719+0000', 'dead_epoch': 122}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:58.973108+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 49625, 'last_purged_snaps_scrub': '2024-04-20T02:52:38.383284+0000', 'dead_epoch': 150}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:29.483129+0000', 'dead_epoch': 26}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:18.346058+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:29.823916+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [6, 0, 2]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 57, 'length': 1}, {'begin': 63, 'length': 1}, {'begin': 65, 'length': 1}, {'begin': 69, 'length': 1}, {'begin': 71, 'length': 1}, {'begin': 75, 'length': 1}, {'begin': 77, 'length': 2}, {'begin': 80, 'length': 1}, {'begin': 83, 'length': 3}, {'begin': 90, 'length': 1}, {'begin': 93, 'length': 1}]}], 'new_removed_snaps': [{'pool': 4, 'snaps': [{'begin': 93, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T02:53:22.611 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:53:22.612 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:22.610+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:22.635 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:22.635 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:22.646 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:22.731 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:22.731+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:22.812 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:22.812+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:22.829 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:22.829 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:22.840 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:22.925 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:22.925+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:22.991 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:53:22.993 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:53:23.004 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-04-20T02:53:22.147441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":234691858,"num_objects":187,"num_object_clones":15,"num_object_copies":561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":1,"num_read":3509,"num_read_kb":3151208,"num_write":2320,"num_write_kb":588597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":270286888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":733508,"kb_used_data":513500,"kb_used_omap":9,"kb_used_meta":211894,"kb_avail":565497532,"statfs":{"total":579820584960,"available":579069472768,"internally_reserved":0,"allocated":525824000,"data_stored":554632209,"data_compressed":10996347,"data_compressed_allocated":12005376,"data_compressed_original":27377664,"omap_allocated":9619,"internal_metadata":216980077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13188527,"num_objects":-13,"num_object_clones":-4,"num_object_copies":-39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13,"num_whiteouts":0,"num_read":-189,"num_read_kb":-250896,"num_write":-21,"num_write_kb":-7209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9,"num_bytes_recovered":-19243742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000870"},"pg_stats":[{"pgid":"4.1","version":"157'133","reported_seq":653,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-20T02:53:17.145592+0000","last_change":"2024-04-20T02:53:17.145592+0000","last_active":"2024-04-20T02:53:17.145592+0000","last_peered":"2024-04-20T02:53:17.145592+0000","last_clean":"2024-04-20T02:53:17.145592+0000","last_became_active":"2024-04-20T02:52:54.150080+0000","last_became_peered":"2024-04-20T02:52:54.150080+0000","last_unstale":"2024-04-20T02:53:17.145592+0000","last_undegraded":"2024-04-20T02:53:17.145592+0000","last_fullsized":"2024-04-20T02:53:17.145592+0000","mapping_epoch":154,"log_start":"137'127","ondisk_log_start":"137'127","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'131","last_scrub_stamp":"2024-04-20T02:52:54.612743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.612743+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":123,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:18.713556+0000","scrub_duration":0.0022862500000000001,"objects_trimmed":0,"snaptrim_duration":0.0014859420000000001,"stat_sum":{"num_bytes":3226977,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":94,"num_read_kb":122130,"num_write":18,"num_write_kb":5336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3980303,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":114,"reported_epoch":178,"state":"peering","last_fresh":"2024-04-20T02:53:17.145228+0000","last_change":"2024-04-20T02:52:21.251209+0000","last_active":"2024-04-20T02:52:20.249100+0000","last_peered":"2024-04-20T02:52:19.239188+0000","last_clean":"2024-04-20T02:52:19.239188+0000","last_became_active":"2024-04-20T02:52:10.211212+0000","last_became_peered":"2024-04-20T02:52:10.211212+0000","last_unstale":"2024-04-20T02:53:17.145228+0000","last_undegraded":"2024-04-20T02:53:17.145228+0000","last_fullsized":"2024-04-20T02:53:17.145228+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:56.983760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.983760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"178'187","reported_seq":692,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:53:17.146512+0000","last_change":"2024-04-20T02:53:17.146512+0000","last_active":"2024-04-20T02:53:17.146512+0000","last_peered":"2024-04-20T02:53:17.146512+0000","last_clean":"2024-04-20T02:53:17.146512+0000","last_became_active":"2024-04-20T02:53:16.152778+0000","last_became_peered":"2024-04-20T02:53:16.152778+0000","last_unstale":"2024-04-20T02:53:17.146512+0000","last_undegraded":"2024-04-20T02:53:17.146512+0000","last_fullsized":"2024-04-20T02:53:17.146512+0000","mapping_epoch":176,"log_start":"172'181","ondisk_log_start":"172'181","created":22,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":177,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001820667,"objects_trimmed":1,"snaptrim_duration":0.002634562,"stat_sum":{"num_bytes":11324980,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":189,"num_read_kb":229535,"num_write":76,"num_write_kb":22594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":16021054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":191,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-20T02:53:16.148171+0000","last_change":"2024-04-20T02:52:51.154350+0000","last_active":"2024-04-20T02:53:16.148171+0000","last_peered":"2024-04-20T02:53:16.148171+0000","last_clean":"2024-04-20T02:53:16.148171+0000","last_became_active":"2024-04-20T02:52:51.153987+0000","last_became_peered":"2024-04-20T02:52:51.153987+0000","last_unstale":"2024-04-20T02:53:16.148171+0000","last_undegraded":"2024-04-20T02:53:16.148171+0000","last_fullsized":"2024-04-20T02:53:16.148171+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:57.514743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:57.514743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001500723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-20T02:53:18.150633+0000","last_change":"2024-04-20T02:52:53.937048+0000","last_active":"2024-04-20T02:53:18.150633+0000","last_peered":"2024-04-20T02:53:18.150633+0000","last_clean":"2024-04-20T02:53:18.150633+0000","last_became_active":"2024-04-20T02:52:51.154174+0000","last_became_peered":"2024-04-20T02:52:51.154174+0000","last_unstale":"2024-04-20T02:53:18.150633+0000","last_undegraded":"2024-04-20T02:53:18.150633+0000","last_fullsized":"2024-04-20T02:53:18.150633+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:53.936957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:53.936957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.656218+0000","scrub_duration":0.0013534090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"174'249","reported_seq":797,"reported_epoch":180,"state":"clean+premerge+peered","last_fresh":"2024-04-20T02:53:19.153818+0000","last_change":"2024-04-20T02:53:19.153818+0000","last_active":"2024-04-20T02:53:18.148639+0000","last_peered":"2024-04-20T02:53:19.153818+0000","last_clean":"2024-04-20T02:53:19.153818+0000","last_became_active":"2024-04-20T02:51:18.502004+0000","last_became_peered":"2024-04-20T02:53:19.153237+0000","last_unstale":"2024-04-20T02:53:19.153818+0000","last_undegraded":"2024-04-20T02:53:19.153818+0000","last_fullsized":"2024-04-20T02:53:19.153818+0000","mapping_epoch":47,"log_start":"161'246","ondisk_log_start":"161'246","created":22,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'243","last_scrub_stamp":"2024-04-20T02:52:54.597931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.597931+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":242,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:04.245801+0000","scrub_duration":0.0028458709999999998,"objects_trimmed":0,"snaptrim_duration":0.00093416500000000004,"stat_sum":{"num_bytes":5630444,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":128,"num_read_kb":196830,"num_write":34,"num_write_kb":10563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"}]},{"pgid":"3.1","version":"180'258","reported_seq":584,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-04-20T02:53:19.190123+0000","last_change":"2024-04-20T02:53:08.766997+0000","last_active":"2024-04-20T02:53:19.190123+0000","last_peered":"2024-04-20T02:53:19.190123+0000","last_clean":"2024-04-20T02:53:19.190123+0000","last_became_active":"2024-04-20T02:52:52.148432+0000","last_became_peered":"2024-04-20T02:52:52.148432+0000","last_unstale":"2024-04-20T02:53:19.190123+0000","last_undegraded":"2024-04-20T02:53:19.190123+0000","last_fullsized":"2024-04-20T02:53:19.190123+0000","mapping_epoch":152,"log_start":"175'255","ondisk_log_start":"175'255","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":280,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:29.132762+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26688310,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":232,"num_read_kb":8229,"num_write":312,"num_write_kb":77096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":78,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-20T02:53:18.149731+0000","last_change":"2024-04-20T02:52:32.918930+0000","last_active":"2024-04-20T02:53:18.149731+0000","last_peered":"2024-04-20T02:53:18.149731+0000","last_clean":"2024-04-20T02:53:18.149731+0000","last_became_active":"2024-04-20T02:52:22.261878+0000","last_became_peered":"2024-04-20T02:52:22.261878+0000","last_unstale":"2024-04-20T02:53:18.149731+0000","last_undegraded":"2024-04-20T02:53:18.149731+0000","last_fullsized":"2024-04-20T02:53:18.149731+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:32.918813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.918813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:36.236915+0000","scrub_duration":0.0014288059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"178'137","reported_seq":621,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:53:17.153550+0000","last_change":"2024-04-20T02:53:17.148572+0000","last_active":"2024-04-20T02:53:17.153550+0000","last_peered":"2024-04-20T02:53:17.153550+0000","last_clean":"2024-04-20T02:53:17.153550+0000","last_became_active":"2024-04-20T02:53:16.151498+0000","last_became_peered":"2024-04-20T02:53:16.151498+0000","last_unstale":"2024-04-20T02:53:17.153550+0000","last_undegraded":"2024-04-20T02:53:17.153550+0000","last_fullsized":"2024-04-20T02:53:17.153550+0000","mapping_epoch":176,"log_start":"162'133","ondisk_log_start":"162'133","created":22,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'57","last_scrub_stamp":"2024-04-20T02:51:21.386796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:21.386796+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":220,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018878359999999999,"objects_trimmed":2,"snaptrim_duration":0.010828997,"stat_sum":{"num_bytes":12371407,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":141,"num_read_kb":161903,"num_write":37,"num_write_kb":9540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22843153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"176'267","reported_seq":569,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-20T02:53:16.250068+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:53:16.250068+0000","last_peered":"2024-04-20T02:53:16.250068+0000","last_clean":"2024-04-20T02:53:16.250068+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:53:16.250068+0000","last_undegraded":"2024-04-20T02:53:16.250068+0000","last_fullsized":"2024-04-20T02:53:16.250068+0000","mapping_epoch":49,"log_start":"174'264","ondisk_log_start":"174'264","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":272,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24109823,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":238,"num_read_kb":13458,"num_write":320,"num_write_kb":91578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":346,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-20T02:53:18.150169+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:53:18.150169+0000","last_peered":"2024-04-20T02:53:18.150169+0000","last_clean":"2024-04-20T02:53:18.150169+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:53:18.150169+0000","last_undegraded":"2024-04-20T02:53:18.150169+0000","last_fullsized":"2024-04-20T02:53:18.150169+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":179,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-20T02:53:17.144287+0000","last_change":"2024-04-20T02:53:08.364116+0000","last_active":"2024-04-20T02:53:17.144287+0000","last_peered":"2024-04-20T02:53:17.144287+0000","last_clean":"2024-04-20T02:53:17.144287+0000","last_became_active":"2024-04-20T02:52:22.265487+0000","last_became_peered":"2024-04-20T02:52:22.265487+0000","last_unstale":"2024-04-20T02:53:17.144287+0000","last_undegraded":"2024-04-20T02:53:17.144287+0000","last_fullsized":"2024-04-20T02:53:17.144287+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.792324+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"161'96","reported_seq":312,"reported_epoch":178,"state":"peering","last_fresh":"2024-04-20T02:53:17.145263+0000","last_change":"2024-04-20T02:53:17.145263+0000","last_active":"2024-04-20T02:53:17.145263+0000","last_peered":"2024-04-20T02:53:00.417446+0000","last_clean":"2024-04-20T02:53:00.417446+0000","last_became_active":"2024-04-20T02:52:54.150076+0000","last_became_peered":"2024-04-20T02:52:54.150076+0000","last_unstale":"2024-04-20T02:53:17.145263+0000","last_undegraded":"2024-04-20T02:53:17.145263+0000","last_fullsized":"2024-04-20T02:53:17.145263+0000","mapping_epoch":178,"log_start":"148'93","ondisk_log_start":"148'93","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'86","last_scrub_stamp":"2024-04-20T02:52:32.901998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.901998+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":123,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:43.974508+0000","scrub_duration":0.0024332220000000001,"objects_trimmed":0,"snaptrim_duration":0.00037001800000000001,"stat_sum":{"num_bytes":6050619,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":21970,"num_write":10,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1543958,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"}]},{"pgid":"3.0","version":"178'234","reported_seq":501,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-20T02:53:17.170403+0000","last_change":"2024-04-20T02:52:42.201016+0000","last_active":"2024-04-20T02:53:17.170403+0000","last_peered":"2024-04-20T02:53:17.170403+0000","last_clean":"2024-04-20T02:53:17.170403+0000","last_became_active":"2024-04-20T02:52:25.272853+0000","last_became_peered":"2024-04-20T02:52:25.272853+0000","last_unstale":"2024-04-20T02:53:17.170403+0000","last_undegraded":"2024-04-20T02:53:17.170403+0000","last_fullsized":"2024-04-20T02:53:17.170403+0000","mapping_epoch":126,"log_start":"178'231","ondisk_log_start":"178'231","created":21,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'188","last_scrub_stamp":"2024-04-20T02:52:42.200865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:52:42.200865+0000","objects_scrubbed":31,"log_size":3,"log_dups_size":243,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.320740+0000","scrub_duration":0.0052191850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20850096,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":192,"num_read_kb":5867,"num_write":284,"num_write_kb":75585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":15696807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-20T02:53:16.154496+0000","last_change":"2024-04-20T02:52:56.212555+0000","last_active":"2024-04-20T02:53:16.154496+0000","last_peered":"2024-04-20T02:53:16.154496+0000","last_clean":"2024-04-20T02:53:16.154496+0000","last_became_active":"2024-04-20T02:52:22.272822+0000","last_became_peered":"2024-04-20T02:52:22.272822+0000","last_unstale":"2024-04-20T02:53:16.154496+0000","last_undegraded":"2024-04-20T02:53:16.154496+0000","last_fullsized":"2024-04-20T02:53:16.154496+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:56.212430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.212430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:20.820102+0000","scrub_duration":0.0013799870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"179'124","reported_seq":686,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-20T02:53:18.145580+0000","last_change":"2024-04-20T02:53:17.146360+0000","last_active":"2024-04-20T02:53:18.145580+0000","last_peered":"2024-04-20T02:53:18.145580+0000","last_clean":"2024-04-20T02:53:18.145580+0000","last_became_active":"2024-04-20T02:52:46.141983+0000","last_became_peered":"2024-04-20T02:52:46.141983+0000","last_unstale":"2024-04-20T02:53:18.145580+0000","last_undegraded":"2024-04-20T02:53:18.145580+0000","last_fullsized":"2024-04-20T02:53:18.145580+0000","mapping_epoch":147,"log_start":"163'121","ondisk_log_start":"163'121","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'119","last_scrub_stamp":"2024-04-20T02:52:56.564632+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.564632+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":307,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:22.552034+0000","scrub_duration":0.0028929759999999998,"objects_trimmed":0,"snaptrim_duration":0.0022418149999999999,"stat_sum":{"num_bytes":3613538,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":29933,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"}]},{"pgid":"3.3","version":"175'232","reported_seq":523,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-20T02:53:18.737801+0000","last_change":"2024-04-20T02:53:18.737801+0000","last_active":"2024-04-20T02:53:18.737801+0000","last_peered":"2024-04-20T02:53:18.737801+0000","last_clean":"2024-04-20T02:53:18.737801+0000","last_became_active":"2024-04-20T02:53:11.150822+0000","last_became_peered":"2024-04-20T02:53:11.150822+0000","last_unstale":"2024-04-20T02:53:18.737801+0000","last_undegraded":"2024-04-20T02:53:18.737801+0000","last_fullsized":"2024-04-20T02:53:18.737801+0000","mapping_epoch":171,"log_start":"174'229","ondisk_log_start":"174'229","created":21,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":235,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.609708+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26691781,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":201,"num_read_kb":6610,"num_write":285,"num_write_kb":74780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":18551655,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":158,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-20T02:53:17.146008+0000","last_change":"2024-04-20T02:51:52.777359+0000","last_active":"2024-04-20T02:53:17.146008+0000","last_peered":"2024-04-20T02:53:17.146008+0000","last_clean":"2024-04-20T02:53:17.146008+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:53:17.146008+0000","last_undegraded":"2024-04-20T02:53:17.146008+0000","last_fullsized":"2024-04-20T02:53:17.146008+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:52.777234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:52.777234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012962270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"169'45","reported_seq":464,"reported_epoch":178,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-20T02:53:17.157375+0000","last_change":"2024-04-20T02:53:17.157375+0000","last_active":"2024-04-20T02:53:17.157375+0000","last_peered":"2024-04-20T02:53:17.157375+0000","last_clean":"2024-04-20T02:53:17.157375+0000","last_became_active":"2024-04-20T02:53:17.156757+0000","last_became_peered":"2024-04-20T02:53:17.156757+0000","last_unstale":"2024-04-20T02:53:17.157375+0000","last_undegraded":"2024-04-20T02:53:17.157375+0000","last_fullsized":"2024-04-20T02:53:17.157375+0000","mapping_epoch":141,"log_start":"169'45","ondisk_log_start":"169'45","created":22,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:52:56.000178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.000178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015078069999999999,"objects_trimmed":0,"snaptrim_duration":0.85353207399999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":26722,"num_write":15,"num_write_kb":3607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5278912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"},{"start":"53","length":"1"}]},{"pgid":"4.2","version":"0'0","reported_seq":223,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-20T02:53:17.146778+0000","last_change":"2024-04-20T02:53:17.146778+0000","last_active":"2024-04-20T02:53:17.146778+0000","last_peered":"2024-04-20T02:53:17.146778+0000","last_clean":"2024-04-20T02:53:17.146778+0000","last_became_active":"2024-04-20T02:52:51.150057+0000","last_became_peered":"2024-04-20T02:52:51.150057+0000","last_unstale":"2024-04-20T02:53:17.146778+0000","last_undegraded":"2024-04-20T02:53:17.146778+0000","last_fullsized":"2024-04-20T02:53:17.146778+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_clean_scrub_stamp":"2024-04-20T02:52:43.686760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:01.323681+0000","scrub_duration":0.0018670150000000001,"objects_trimmed":0,"snaptrim_duration":0.00087645699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":182,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-04-20T02:53:16.147738+0000","last_change":"2024-04-20T02:52:18.363613+0000","last_active":"2024-04-20T02:53:16.147738+0000","last_peered":"2024-04-20T02:53:16.147738+0000","last_clean":"2024-04-20T02:53:16.147738+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:53:16.147738+0000","last_undegraded":"2024-04-20T02:53:16.147738+0000","last_fullsized":"2024-04-20T02:53:16.147738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:18.363433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:18.363433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:38.745528+0000","scrub_duration":0.0016555929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"175'168","reported_seq":647,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-20T02:53:18.349628+0000","last_change":"2024-04-20T02:53:17.145791+0000","last_active":"2024-04-20T02:53:18.349628+0000","last_peered":"2024-04-20T02:53:18.349628+0000","last_clean":"2024-04-20T02:53:18.349628+0000","last_became_active":"2024-04-20T02:52:13.145577+0000","last_became_peered":"2024-04-20T02:52:13.145577+0000","last_unstale":"2024-04-20T02:53:18.349628+0000","last_undegraded":"2024-04-20T02:53:18.349628+0000","last_fullsized":"2024-04-20T02:53:18.349628+0000","mapping_epoch":62,"log_start":"163'164","ondisk_log_start":"163'164","created":22,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'74","last_scrub_stamp":"2024-04-20T02:51:56.784299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.784299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":322,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015920019999999999,"objects_trimmed":0,"snaptrim_duration":0.0017537329999999999,"stat_sum":{"num_bytes":9714578,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":103552,"num_write":41,"num_write_kb":10973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"}]},{"pgid":"4.9","version":"177'472","reported_seq":1079,"reported_epoch":178,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-20T02:53:17.179163+0000","last_change":"2024-04-20T02:53:17.179210+0000","last_active":"2024-04-20T02:53:17.179163+0000","last_peered":"2024-04-20T02:53:17.179163+0000","last_clean":"2024-04-20T02:53:17.179163+0000","last_became_active":"2024-04-20T02:53:11.151224+0000","last_became_peered":"2024-04-20T02:53:11.151224+0000","last_unstale":"2024-04-20T02:53:17.179163+0000","last_undegraded":"2024-04-20T02:53:17.179163+0000","last_fullsized":"2024-04-20T02:53:17.179163+0000","mapping_epoch":171,"log_start":"175'468","ondisk_log_start":"175'468","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'294","last_scrub_stamp":"2024-04-20T02:52:16.813424+0000","last_deep_scrub":"98'280","last_deep_scrub_stamp":"2024-04-20T02:51:56.947305+0000","last_clean_scrub_stamp":"2024-04-20T02:52:16.813424+0000","objects_scrubbed":7,"log_size":4,"log_dups_size":1089,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:41.845382+0000","scrub_duration":0.0034018329999999999,"objects_trimmed":0,"snaptrim_duration":0.032058398000000002,"stat_sum":{"num_bytes":33605128,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":134,"num_read_kb":127940,"num_write":93,"num_write_kb":27243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":34386920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"}]},{"pgid":"4.a","version":"177'116","reported_seq":536,"reported_epoch":178,"state":"peering","last_fresh":"2024-04-20T02:53:17.146236+0000","last_change":"2024-04-20T02:53:17.146236+0000","last_active":"2024-04-20T02:53:17.146236+0000","last_peered":"2024-04-20T02:53:16.225834+0000","last_clean":"2024-04-20T02:53:16.225834+0000","last_became_active":"2024-04-20T02:52:10.138005+0000","last_became_peered":"2024-04-20T02:52:10.138005+0000","last_unstale":"2024-04-20T02:53:17.146236+0000","last_undegraded":"2024-04-20T02:53:17.146236+0000","last_fullsized":"2024-04-20T02:53:17.146236+0000","mapping_epoch":178,"log_start":"170'112","ondisk_log_start":"170'112","created":22,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'53","last_scrub_stamp":"2024-04-20T02:52:05.810627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:05.810627+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":112,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:28.333446+0000","scrub_duration":0.0022334680000000002,"objects_trimmed":1,"snaptrim_duration":0.85224467400000004,"stat_sum":{"num_bytes":4655605,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":192,"num_read_kb":217665,"num_write":60,"num_write_kb":15678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"},{"start":"55","length":"1"}]},{"pgid":"4.b","version":"179'344","reported_seq":876,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-04-20T02:53:18.433192+0000","last_change":"2024-04-20T02:53:17.149747+0000","last_active":"2024-04-20T02:53:18.433192+0000","last_peered":"2024-04-20T02:53:18.433192+0000","last_clean":"2024-04-20T02:53:18.433192+0000","last_became_active":"2024-04-20T02:53:11.147630+0000","last_became_peered":"2024-04-20T02:53:11.147630+0000","last_unstale":"2024-04-20T02:53:18.433192+0000","last_undegraded":"2024-04-20T02:53:18.433192+0000","last_fullsized":"2024-04-20T02:53:18.433192+0000","mapping_epoch":171,"log_start":"179'341","ondisk_log_start":"179'341","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'157","last_scrub_stamp":"2024-04-20T02:52:07.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:07.013316+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":341,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:24.431814+0000","scrub_duration":0.002193338,"objects_trimmed":0,"snaptrim_duration":0.00073235499999999996,"stat_sum":{"num_bytes":12373752,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":370,"num_read_kb":444658,"num_write":174,"num_write_kb":47563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":23150965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"}]},{"pgid":"4.c","version":"178'159","reported_seq":601,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:53:17.237679+0000","last_change":"2024-04-20T02:53:17.237679+0000","last_active":"2024-04-20T02:53:17.237679+0000","last_peered":"2024-04-20T02:53:17.237679+0000","last_clean":"2024-04-20T02:53:17.237679+0000","last_became_active":"2024-04-20T02:53:11.148476+0000","last_became_peered":"2024-04-20T02:53:11.148476+0000","last_unstale":"2024-04-20T02:53:17.237679+0000","last_undegraded":"2024-04-20T02:53:17.237679+0000","last_fullsized":"2024-04-20T02:53:17.237679+0000","mapping_epoch":171,"log_start":"173'155","ondisk_log_start":"173'155","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'95","last_scrub_stamp":"2024-04-20T02:52:09.803923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:09.803923+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":155,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:32.262327+0000","scrub_duration":0.0026115750000000001,"objects_trimmed":1,"snaptrim_duration":0.093435299999999999,"stat_sum":{"num_bytes":4773320,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":160,"num_read_kb":171153,"num_write":90,"num_write_kb":24779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":14051707,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"}]},{"pgid":"4.d","version":"178'279","reported_seq":842,"reported_epoch":178,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:53:17.179332+0000","last_change":"2024-04-20T02:53:17.179332+0000","last_active":"2024-04-20T02:53:17.179332+0000","last_peered":"2024-04-20T02:53:17.179332+0000","last_clean":"2024-04-20T02:53:17.179332+0000","last_became_active":"2024-04-20T02:52:46.144032+0000","last_became_peered":"2024-04-20T02:52:46.144032+0000","last_unstale":"2024-04-20T02:53:17.179332+0000","last_undegraded":"2024-04-20T02:53:17.179332+0000","last_fullsized":"2024-04-20T02:53:17.179332+0000","mapping_epoch":147,"log_start":"172'275","ondisk_log_start":"172'275","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'169","last_scrub_stamp":"2024-04-20T02:52:08.841360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:08.841360+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":275,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:23.317619+0000","scrub_duration":0.002314502,"objects_trimmed":2,"snaptrim_duration":0.077905163999999999,"stat_sum":{"num_bytes":8773386,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":418,"num_read_kb":513736,"num_write":159,"num_write_kb":42985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":27793283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"},{"start":"50","length":"1"},{"start":"53","length":"1"}]},{"pgid":"4.e","version":"175'120","reported_seq":575,"reported_epoch":175,"state":"peering","last_fresh":"2024-04-20T02:53:15.172652+0000","last_change":"2024-04-20T02:53:17.146932+0000","last_active":"2024-04-20T02:53:17.146932+0000","last_peered":"2024-04-20T02:53:15.172652+0000","last_clean":"2024-04-20T02:53:15.172652+0000","last_became_active":"2024-04-20T02:53:11.146801+0000","last_became_peered":"2024-04-20T02:53:11.146801+0000","last_unstale":"2024-04-20T02:53:15.172652+0000","last_undegraded":"2024-04-20T02:53:15.172652+0000","last_fullsized":"2024-04-20T02:53:15.172652+0000","mapping_epoch":178,"log_start":"172'116","ondisk_log_start":"172'116","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'77","last_scrub_stamp":"2024-04-20T02:51:46.088382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:46.088382+0000","objects_scrubbed":3,"log_size":4,"log_dups_size":116,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002170693,"objects_trimmed":1,"snaptrim_duration":0.95365011899999996,"stat_sum":{"num_bytes":6269110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":266051,"num_write":58,"num_write_kb":18271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":10957139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"1"}]},{"pgid":"4.f","version":"163'44","reported_seq":452,"reported_epoch":177,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-20T02:53:16.225798+0000","last_change":"2024-04-20T02:53:16.225798+0000","last_active":"2024-04-20T02:53:16.225798+0000","last_peered":"2024-04-20T02:53:16.225798+0000","last_clean":"2024-04-20T02:53:16.225798+0000","last_became_active":"2024-04-20T02:52:54.156070+0000","last_became_peered":"2024-04-20T02:52:54.156070+0000","last_unstale":"2024-04-20T02:53:16.225798+0000","last_undegraded":"2024-04-20T02:53:16.225798+0000","last_fullsized":"2024-04-20T02:53:16.225798+0000","mapping_epoch":154,"log_start":"153'40","ondisk_log_start":"153'40","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'36","last_scrub_stamp":"2024-04-20T02:52:46.138400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:46.138400+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":40,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:09.709596+0000","scrub_duration":0.003388247,"objects_trimmed":0,"snaptrim_duration":0.078219753000000003,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":115234,"num_write":27,"num_write_kb":7507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1714357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"}]},{"pgid":"4.10","version":"173'137","reported_seq":768,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-20T02:53:17.145534+0000","last_change":"2024-04-20T02:53:17.145914+0000","last_active":"2024-04-20T02:53:17.145534+0000","last_peered":"2024-04-20T02:53:17.145534+0000","last_clean":"2024-04-20T02:53:17.145534+0000","last_became_active":"2024-04-20T02:52:46.140881+0000","last_became_peered":"2024-04-20T02:52:46.140881+0000","last_unstale":"2024-04-20T02:53:17.145534+0000","last_undegraded":"2024-04-20T02:53:17.145534+0000","last_fullsized":"2024-04-20T02:53:17.145534+0000","mapping_epoch":147,"log_start":"157'134","ondisk_log_start":"157'134","created":60,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'132","last_scrub_stamp":"2024-04-20T02:52:55.175686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:55.175686+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":132,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.746787+0000","scrub_duration":0.0022427829999999999,"objects_trimmed":0,"snaptrim_duration":0.0021710240000000001,"stat_sum":{"num_bytes":6403533,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":203221,"num_write":25,"num_write_kb":7645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":17455143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"}]},{"pgid":"4.11","version":"178'232","reported_seq":941,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-20T02:53:17.197222+0000","last_change":"2024-04-20T02:53:17.146215+0000","last_active":"2024-04-20T02:53:17.197222+0000","last_peered":"2024-04-20T02:53:17.197222+0000","last_clean":"2024-04-20T02:53:17.197222+0000","last_became_active":"2024-04-20T02:53:11.146921+0000","last_became_peered":"2024-04-20T02:53:11.146921+0000","last_unstale":"2024-04-20T02:53:17.197222+0000","last_undegraded":"2024-04-20T02:53:17.197222+0000","last_fullsized":"2024-04-20T02:53:17.197222+0000","mapping_epoch":171,"log_start":"176'229","ondisk_log_start":"176'229","created":60,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":5,"last_scrub":"176'229","last_scrub_stamp":"2024-04-20T02:53:16.347958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.347958+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":229,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:31.654441+0000","scrub_duration":0.002308816,"objects_trimmed":0,"snaptrim_duration":0.0019002349999999999,"stat_sum":{"num_bytes":4857355,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":139,"num_read_kb":164711,"num_write":43,"num_write_kb":10474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":24126526,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"}]},{"pgid":"4.12","version":"0'0","reported_seq":369,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-04-20T02:53:17.146739+0000","last_change":"2024-04-20T02:53:17.146739+0000","last_active":"2024-04-20T02:53:17.146739+0000","last_peered":"2024-04-20T02:53:17.146739+0000","last_clean":"2024-04-20T02:53:17.146739+0000","last_became_active":"2024-04-20T02:52:51.151032+0000","last_became_peered":"2024-04-20T02:52:51.151032+0000","last_unstale":"2024-04-20T02:53:17.146739+0000","last_undegraded":"2024-04-20T02:53:17.146739+0000","last_fullsized":"2024-04-20T02:53:17.146739+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:07.368016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:07.368016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:30.123817+0000","scrub_duration":0.003090041,"objects_trimmed":0,"snaptrim_duration":0.0011225950000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":19,"stat_sum":{"num_bytes":135761480,"num_objects":54,"num_object_clones":15,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":1,"num_read":2530,"num_read_kb":3116944,"num_write":960,"num_write_kb":267142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":203303420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358531072,"data_stored":389851529,"data_compressed":462641,"data_compressed_allocated":434176,"data_compressed_original":1179648,"omap_allocated":2595,"internal_metadata":0},"log_size":62,"ondisk_log_size":62,"up":57,"acting":57,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98340010,"num_objects":131,"num_object_clones":0,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":863,"num_read_kb":34164,"num_write":1201,"num_write_kb":319039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":66983468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":226611200,"data_stored":241819292,"data_compressed":12362124,"data_compressed_allocated":13398016,"data_compressed_original":29306880,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509482,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83088,"kb_used_data":47876,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288752,"statfs":{"total":96636764160,"available":96551682048,"internally_reserved":0,"allocated":49025024,"data_stored":52905553,"data_compressed":69385,"data_compressed_allocated":102400,"data_compressed_original":708608,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.442,"15min":0.442},"min":{"1min":0.27000000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57699999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.45900000000000002},"min":{"1min":0.32400000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.42399999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.53200000000000003,"15min":0.47799999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.9369999999999994,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.68500000000000005,"15min":0.64800000000000002},"min":{"1min":0.32400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":9.8119999999999994,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.443},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.46400000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.61399999999999999,"15min":0.56200000000000006},"min":{"1min":0.35499999999999998,"5min":0.307,"15min":0.307},"max":{"1min":9.6440000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.313},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.56399999999999995,"15min":0.55000000000000004},"min":{"1min":0.29999999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.8599999999999999,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.34000000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.435,"15min":0.443},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65500000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.59399999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.441,"15min":0.45000000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57699999999999996,"5min":2.431,"15min":2.431},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.51500000000000001,"15min":0.52200000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.64600000000000002,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73299999999999998,"5min":1.599,"15min":1.599},"last":0.54700000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108724,"kb_used_data":71512,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94263116,"statfs":{"total":96636764160,"available":96525430784,"internally_reserved":0,"allocated":73228288,"data_stored":74102837,"data_compressed":1867380,"data_compressed_allocated":2027520,"data_compressed_original":4362240,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56599999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.37}]},{"osd":1,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42299999999999999,"15min":0.41699999999999998},"min":{"1min":0.34699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.441},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.47899999999999998,"15min":0.46100000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.437,"15min":0.434},"min":{"1min":0.34499999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.32900000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.27900000000000003,"5min":0.252,"15min":0.252},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.41799999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.441,"15min":0.437},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.432},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.39400000000000002}]}]},{"osd":2,"up_from":0,"seq":592705486858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147056,"kb_used_data":106420,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94224784,"statfs":{"total":96636764160,"available":96486178816,"internally_reserved":0,"allocated":108974080,"data_stored":115706981,"data_compressed":1792751,"data_compressed_allocated":1941504,"data_compressed_original":4251648,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.30199999999999999,"5min":0.34200000000000003,"15min":0.34799999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.42399999999999999,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.23100000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.251},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.27000000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.40799999999999997,"15min":0.41799999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.58999999999999997,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39700000000000002,"15min":0.39900000000000002},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.94299999999999995,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.35199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.753,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.38700000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.63600000000000001,"5min":1.097,"15min":1.097},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.53500000000000003,"15min":0.54900000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.56399999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.31}]},{"osd":6,"last update":"Sat Apr 20 02:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35999999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73599999999999999,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.372},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60099999999999998,"15min":0.58999999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":3.2719999999999998,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.36099999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.72699999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58599999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.77900000000000003,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.39500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123036,"kb_used_data":87816,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94248804,"statfs":{"total":96636764160,"available":96510775296,"internally_reserved":0,"allocated":89923584,"data_stored":97281692,"data_compressed":1857527,"data_compressed_allocated":2015232,"data_compressed_original":4362240,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.29299999999999998,"5min":0.29799999999999999,"15min":0.29899999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52500000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.377,"15min":0.38100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57999999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.33600000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.33300000000000002,"15min":0.33800000000000002},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.48399999999999999,"5min":0.751,"15min":0.751},"last":0.223},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.373,"15min":0.379},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.496,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.27900000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":1.198,"15min":1.198},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.32200000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.85899999999999999,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.52900000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.01,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.36299999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.376},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47199999999999998,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69899999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.40699999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.54300000000000004,"15min":0.55500000000000005},"min":{"1min":0.34699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64100000000000001,"5min":1.153,"15min":1.153},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.42699999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.56399999999999995,"15min":0.57199999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90700000000000003,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.47399999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148260,"kb_used_data":113040,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94223580,"statfs":{"total":96636764160,"available":96484945920,"internally_reserved":0,"allocated":115752960,"data_stored":119124907,"data_compressed":2731405,"data_compressed_allocated":2998272,"data_compressed_original":7012352,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.78700000000000003,"15min":0.84699999999999998},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.46000000000000002,"15min":0.439},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48699999999999999,"15min":0.48399999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.0680000000000001,"5min":1.419,"15min":1.419},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.58899999999999997,"15min":0.59899999999999998},"min":{"1min":0.32600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.756,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.442}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.69499999999999995,"15min":0.73499999999999999},"min":{"1min":0.372,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.59699999999999998,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.80800000000000005,"15min":0.85699999999999998},"min":{"1min":0.35499999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.1279999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.43099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.625,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.40699999999999997}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.57799999999999996,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.66100000000000003,"5min":1.038,"15min":1.038},"last":0.45000000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.47999999999999998,"15min":0.46899999999999997},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.59799999999999998,"15min":0.624},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68000000000000005,"5min":3.98,"15min":3.98},"last":0.46000000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509478,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123344,"kb_used_data":86836,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94248496,"statfs":{"total":96636764160,"available":96510459904,"internally_reserved":0,"allocated":88920064,"data_stored":95510239,"data_compressed":2677899,"data_compressed_allocated":2920448,"data_compressed_original":6680576,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36399999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.377,"15min":0.38200000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.81799999999999995,"5min":1.427,"15min":1.427},"last":0.25700000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.36499999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.42799999999999999,"15min":0.41199999999999998},"min":{"1min":0.23899999999999999,"5min":0.215,"15min":0.215},"max":{"1min":3.2730000000000001,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":0.311}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90500000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.497}]},{"osd":5,"last update":"Sat Apr 20 02:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69599999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.627}]},{"osd":6,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.5349999999999999,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.55400000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.878,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.68799999999999994,"15min":0.70899999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3380000000000001,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.50800000000000001}]}]},{"osd":4,"up_from":0,"seq":717259538436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49459200,"data_stored":52559301,"data_compressed":2584080,"data_compressed_allocated":2793472,"data_compressed_original":6037504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32587776,"data_stored":34695797,"data_compressed":1695693,"data_compressed_allocated":1839104,"data_compressed_original":4055040,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12894208,"data_stored":13909415,"data_compressed":734140,"data_compressed_allocated":790528,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32776192,"data_stored":34972751,"data_compressed":1846001,"data_compressed_allocated":2002944,"data_compressed_original":4296704,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18509824,"data_stored":19818665,"data_compressed":1037033,"data_compressed_allocated":1118208,"data_compressed_original":2482176,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32780288,"data_stored":34975823,"data_compressed":1846001,"data_compressed_allocated":2002944,"data_compressed_original":4296704,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":47603712,"data_stored":50887540,"data_compressed":2619176,"data_compressed_allocated":2850816,"data_compressed_original":6303744,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36581376,"data_stored":40611266,"data_compressed":54370,"data_compressed_allocated":57344,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68689920,"data_stored":74538071,"data_compressed":97058,"data_compressed_allocated":102400,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2595,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53215232,"data_stored":58650104,"data_compressed":11526,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56590336,"data_stored":67896718,"data_compressed":95804,"data_compressed_allocated":53248,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":38526976,"data_stored":38242571,"data_compressed":21379,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":60092416,"data_stored":60001051,"data_compressed":106911,"data_compressed_allocated":114688,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":44834816,"data_stored":49911748,"data_compressed":75593,"data_compressed_allocated":69632,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:53:23.007 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [0, 7, 4, 3, 1, 2] 2024-04-20T02:53:23.007 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.0', '0', '7', '4', '3', '1', '2'] 2024-04-20T02:53:23.007 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.0 0 7 4 3 1 2 2024-04-20T02:53:23.011 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:23.011 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:23.015 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:23.014+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:23.023 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:23.116 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:23.117+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:23.154 INFO:tasks.rados.rados.0.smithi105.stdout:2001: done (11 left) 2024-04-20T02:53:23.154 INFO:tasks.rados.rados.0.smithi105.stdout:2002: write oid 37 current snap is 48 2024-04-20T02:53:23.154 INFO:tasks.rados.rados.0.smithi105.stdout:2002: seq_num 414 ranges {558098=636148,1826604=665129} 2024-04-20T02:53:23.158 INFO:tasks.rados.rados.0.smithi105.stdout:2002: writing smithi10548823-37 from 558098 to 1194246 tid 1 2024-04-20T02:53:23.166 INFO:tasks.rados.rados.0.smithi105.stdout:2002: writing smithi10548823-37 from 1826604 to 2491733 tid 2 2024-04-20T02:53:23.166 INFO:tasks.rados.rados.0.smithi105.stdout:2003: tier_promote oid 26 2024-04-20T02:53:23.166 INFO:tasks.rados.rados.0.smithi105.stdout:2004: set_chunk oid 29 target oid 2024-04-20T02:53:23.207 INFO:tasks.rados.rados.0.smithi105.stdout:2004: set_chunk oid 29 offset: 459776 length: 1173504 target oid e2349f2c2d30d5d60b544d45cfeb9c28905f67e62d836f9c017ba6ef0d05e3ff offset: 0 snap -1 2024-04-20T02:53:23.207 INFO:tasks.rados.rados.0.smithi105.stdout:2005: read oid 30 snap -1 2024-04-20T02:53:23.207 INFO:tasks.rados.rados.0.smithi105.stdout:2005: expect deleted 2024-04-20T02:53:23.207 INFO:tasks.rados.rados.0.smithi105.stdout:2006: tier_promote oid 11 2024-04-20T02:53:23.207 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 16 2024-04-20T02:53:23.207 INFO:tasks.rados.rados.0.smithi105.stdout:1993: expect (ObjNum 311 snap 33 seq_num 311) 2024-04-20T02:53:23.217 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:23.217+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:23.219 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:23.219 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:23.230 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:23.318 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:23.318+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:23.378 INFO:tasks.rados.rados.0.smithi105.stdout:1956: expect (ObjNum 333 snap 36 seq_num 333) 2024-04-20T02:53:23.399 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:23.399 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:23.410 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:23.452 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:23.452+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:23.498 INFO:tasks.rados.rados.0.smithi105.stdout:1971: finishing write tid 1 to smithi10548823-7 2024-04-20T02:53:23.498 INFO:tasks.rados.rados.0.smithi105.stdout:1971: finishing write tid 2 to smithi10548823-7 2024-04-20T02:53:23.498 INFO:tasks.rados.rados.0.smithi105.stdout:1971: finishing write tid 3 to smithi10548823-7 2024-04-20T02:53:23.498 INFO:tasks.rados.rados.0.smithi105.stdout:1956: done (15 left) 2024-04-20T02:53:23.498 INFO:tasks.rados.rados.0.smithi105.stdout:1992: done (14 left) 2024-04-20T02:53:23.498 INFO:tasks.rados.rados.0.smithi105.stdout:1993: done (13 left) 2024-04-20T02:53:23.498 INFO:tasks.rados.rados.0.smithi105.stdout:2007: snap_create 2024-04-20T02:53:23.498 INFO:tasks.rados.rados.0.smithi105.stdout:1971: finishing write tid 4 to smithi10548823-7 2024-04-20T02:53:23.498 INFO:tasks.rados.rados.0.smithi105.stdout:1971: finishing write tid 5 to smithi10548823-7 2024-04-20T02:53:23.498 INFO:tasks.rados.rados.0.smithi105.stdout:1971: oid 7 updating version 0 to 296 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1971: oid 7 updating version 296 to 298 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1971: oid 7 updating version 298 to 299 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1971: oid 7 version 299 is already newer than 297 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 7 v 299 (ObjNum 405 snap 46 seq_num 405) dirty exists 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1971: left oid 7 (ObjNum 405 snap 46 seq_num 405) 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1983: finishing write tid 1 to smithi10548823-1 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1983: finishing write tid 2 to smithi10548823-1 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1983: finishing write tid 3 to smithi10548823-1 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1983: finishing write tid 4 to smithi10548823-1 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1983: finishing write tid 5 to smithi10548823-1 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1983: oid 1 updating version 0 to 302 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1983: oid 1 updating version 302 to 304 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1983: oid 1 version 304 is already newer than 303 2024-04-20T02:53:23.499 INFO:tasks.rados.rados.0.smithi105.stdout:1983: oid 1 version 304 is already newer than 301 2024-04-20T02:53:23.500 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 304 (ObjNum 410 snap 46 seq_num 410) dirty exists 2024-04-20T02:53:23.500 INFO:tasks.rados.rados.0.smithi105.stdout:1983: left oid 1 (ObjNum 410 snap 46 seq_num 410) 2024-04-20T02:53:23.500 INFO:tasks.rados.rados.0.smithi105.stdout:1999: got (0) Success 2024-04-20T02:53:23.500 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 22 v 126 (ObjNum 361 snap 37 seq_num 361) dirty exists 2024-04-20T02:53:23.500 INFO:tasks.rados.rados.0.smithi105.stdout:1996: finishing rollback tid 1 to smithi10548823-16 2024-04-20T02:53:23.500 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 16 v 0 (ObjNum 129 snap 0 seq_num 58996112) dirty dne 2024-04-20T02:53:23.500 INFO:tasks.rados.rados.0.smithi105.stdout:1998: expect (ObjNum 400 snap 46 seq_num 400) 2024-04-20T02:53:23.519 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:23.519+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:23.582 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:23.582 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:23.589 INFO:tasks.rados.rados.0.smithi105.stdout:1997: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:53:23.593 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:23.635 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:23.635+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:23.721 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:23.721+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:23.766 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:23.766 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:23.774 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:1995: finishing write tid 1 to smithi10548823-20 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:1995: finishing write tid 2 to smithi10548823-20 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:1995: finishing write tid 3 to smithi10548823-20 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:1995: finishing write tid 4 to smithi10548823-20 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:1995: oid 20 updating version 0 to 346 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:1995: oid 20 updating version 346 to 347 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:1995: oid 20 updating version 347 to 348 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 20 v 348 (ObjNum 413 snap 48 seq_num 413) dirty exists 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:1995: left oid 20 (ObjNum 413 snap 48 seq_num 413) 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:2000: finishing set_chunk to oid 35 2024-04-20T02:53:23.799 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 204288 length: 1084416 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 35 v 0 (ObjNum 124 snap 9 seq_num 124) dirty exists 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:2003: got (0) Success 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 26 v 485 (ObjNum 408 snap 46 seq_num 408) dirty exists 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:2004: finishing set_chunk to oid 29 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 29 set_chunk returned error code -95 offset: 459776 length: 1173504 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 29 v 0 (ObjNum 364 snap 37 seq_num 364) dirty exists 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:2002: finishing write tid 1 to smithi10548823-37 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:2006: got (2) No such file or directory 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:2006: got expected ENOENT (src dne) 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 193 (ObjNum 177 snap 0 seq_num 59248560) clean dne 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:2002: finishing write tid 2 to smithi10548823-37 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:2002: finishing write tid 3 to smithi10548823-37 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:2002: finishing write tid 4 to smithi10548823-37 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:2002: oid 37 updating version 0 to 118 2024-04-20T02:53:23.800 INFO:tasks.rados.rados.0.smithi105.stdout:2002: oid 37 updating version 118 to 119 2024-04-20T02:53:23.801 INFO:tasks.rados.rados.0.smithi105.stdout:2002: oid 37 updating version 119 to 120 2024-04-20T02:53:23.801 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 37 v 120 (ObjNum 414 snap 48 seq_num 414) dirty exists 2024-04-20T02:53:23.801 INFO:tasks.rados.rados.0.smithi105.stdout:2002: left oid 37 (ObjNum 414 snap 48 seq_num 414) 2024-04-20T02:53:23.822 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:23.822+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:23.902 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:23.903 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:23.913 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:23.922 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:23.923+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:24.024 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:24.023+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:24.081 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:24.081 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:24.091 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:24.135 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:24.135+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:24.149 INFO:tasks.rados.rados.0.smithi105.stdout:1971: done (13 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:1983: done (12 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:1995: done (11 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:1996: done (10 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:1997: done (9 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:1998: done (8 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:1999: done (7 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:2000: done (6 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:2002: done (5 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:2003: done (4 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:2004: done (3 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:2005: done (2 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:2006: done (1 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:2007: done (0 left) 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:2008: read oid 1 snap -1 2024-04-20T02:53:24.150 INFO:tasks.rados.rados.0.smithi105.stdout:2008: expect (ObjNum 410 snap 46 seq_num 410) 2024-04-20T02:53:24.151 INFO:tasks.rados.rados.0.smithi105.stdout:2009: read oid 13 snap -1 2024-04-20T02:53:24.151 INFO:tasks.rados.rados.0.smithi105.stdout:2009: expect (ObjNum 412 snap 46 seq_num 412) 2024-04-20T02:53:24.151 INFO:tasks.rados.rados.0.smithi105.stdout:2010: write oid 48 current snap is 49 2024-04-20T02:53:24.151 INFO:tasks.rados.rados.0.smithi105.stdout:2010: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-04-20T02:53:24.156 INFO:tasks.rados.rados.0.smithi105.stdout:2010: writing smithi10548823-48 from 574905 to 1286302 tid 1 2024-04-20T02:53:24.160 INFO:teuthology.orchestra.run.smithi105.stderr:set 2.0 pg_upmap_items mapping to [0->7,4->3,1->2] 2024-04-20T02:53:24.163 INFO:tasks.rados.rados.0.smithi105.stdout:2010: writing smithi10548823-48 from 1885086 to 2410226 tid 2 2024-04-20T02:53:24.167 INFO:tasks.rados.rados.0.smithi105.stdout:2010: writing smithi10548823-48 from 2930189 to 2974905 tid 3 2024-04-20T02:53:24.167 INFO:tasks.rados.rados.0.smithi105.stdout:2011: read oid 21 snap 48 2024-04-20T02:53:24.167 INFO:tasks.rados.rados.0.smithi105.stdout:2011: expect (ObjNum 382 snap 41 seq_num 382) 2024-04-20T02:53:24.167 INFO:tasks.rados.rados.0.smithi105.stdout:2012: read oid 33 snap -1 2024-04-20T02:53:24.168 INFO:tasks.rados.rados.0.smithi105.stdout:2012: expect deleted 2024-04-20T02:53:24.168 INFO:tasks.rados.rados.0.smithi105.stdout:2013: write oid 12 current snap is 49 2024-04-20T02:53:24.168 INFO:tasks.rados.rados.0.smithi105.stdout:2013: seq_num 416 ranges {591712=786646,1943568=668798} 2024-04-20T02:53:24.169 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:53:24.176 INFO:tasks.rados.rados.0.smithi105.stdout:2013: writing smithi10548823-12 from 591712 to 1378358 tid 1 2024-04-20T02:53:24.184 INFO:tasks.rados.rados.0.smithi105.stdout:2013: writing smithi10548823-12 from 1943568 to 2612366 tid 2 2024-04-20T02:53:24.184 INFO:tasks.rados.rados.0.smithi105.stdout:2014: read oid 44 snap -1 2024-04-20T02:53:24.184 INFO:tasks.rados.rados.0.smithi105.stdout:2014: expect (ObjNum 390 snap 44 seq_num 390) 2024-04-20T02:53:24.195 INFO:tasks.rados.rados.0.smithi105.stdout:2015: rollback oid 11 current snap is 49 2024-04-20T02:53:24.195 INFO:tasks.rados.rados.0.smithi105.stdout:rollback oid 11 to 47 2024-04-20T02:53:24.195 INFO:tasks.rados.rados.0.smithi105.stdout:2016: snap_create 2024-04-20T02:53:24.195 INFO:tasks.rados.rados.0.smithi105.stdout:2008: expect (ObjNum 410 snap 46 seq_num 410) 2024-04-20T02:53:24.224 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:24.224+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:24.240 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:24.240 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:24.251 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:24.325 INFO:tasks.rados.rados.0.smithi105.stdout:2009: expect (ObjNum 412 snap 46 seq_num 412) 2024-04-20T02:53:24.325 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:24.325+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:24.396 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:24.422 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:24.422 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:24.426 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:24.426+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:24.432 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:24.498 INFO:tasks.rados.rados.0.smithi105.stdout:2011: expect (ObjNum 382 snap 41 seq_num 382) 2024-04-20T02:53:24.527 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:24.527+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:24.543 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:24.562 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:24.581 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:24.600 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:24.602 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:24.602 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:24.613 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:24.619 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:24.639 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:24.653 INFO:tasks.rados.rados.0.smithi105.stdout:2015: finishing rollback tid 1 to smithi10548823-11 2024-04-20T02:53:24.653 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 11 v 0 (ObjNum 177 snap 0 seq_num 58942624) dirty dne 2024-04-20T02:53:24.653 INFO:tasks.rados.rados.0.smithi105.stdout:2014: expect (ObjNum 390 snap 44 seq_num 390) 2024-04-20T02:53:24.656 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:24.656+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:24.658 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:24.677 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:24.679 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:24.679 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:24.679 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:24.679 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:24.679 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:24.679 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:24.679 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:24.679 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:24.731 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:24.731+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:24.743 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:24.743 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:24.744 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:24.744 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:24.754 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:24.783 INFO:tasks.rados.rados.0.smithi105.stdout:2010: finishing write tid 1 to smithi10548823-48 2024-04-20T02:53:24.783 INFO:tasks.rados.rados.0.smithi105.stdout:2010: finishing write tid 2 to smithi10548823-48 2024-04-20T02:53:24.783 INFO:tasks.rados.rados.0.smithi105.stdout:2010: finishing write tid 3 to smithi10548823-48 2024-04-20T02:53:24.783 INFO:tasks.rados.rados.0.smithi105.stdout:2010: finishing write tid 4 to smithi10548823-48 2024-04-20T02:53:24.783 INFO:tasks.rados.rados.0.smithi105.stdout:2010: finishing write tid 5 to smithi10548823-48 2024-04-20T02:53:24.783 INFO:tasks.rados.rados.0.smithi105.stdout:2010: oid 48 updating version 0 to 146 2024-04-20T02:53:24.783 INFO:tasks.rados.rados.0.smithi105.stdout:2010: oid 48 updating version 146 to 147 2024-04-20T02:53:24.783 INFO:tasks.rados.rados.0.smithi105.stdout:2010: oid 48 updating version 147 to 148 2024-04-20T02:53:24.783 INFO:tasks.rados.rados.0.smithi105.stdout:2010: oid 48 updating version 148 to 149 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 149 (ObjNum 415 snap 49 seq_num 415) dirty exists 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:2010: left oid 48 (ObjNum 415 snap 49 seq_num 415) 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:2013: finishing write tid 1 to smithi10548823-12 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:2013: finishing write tid 2 to smithi10548823-12 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:2013: finishing write tid 3 to smithi10548823-12 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:2013: finishing write tid 4 to smithi10548823-12 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:2013: oid 12 updating version 0 to 46 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:2013: oid 12 updating version 46 to 47 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:2013: oid 12 updating version 47 to 48 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 12 v 48 (ObjNum 416 snap 49 seq_num 416) dirty exists 2024-04-20T02:53:24.784 INFO:tasks.rados.rados.0.smithi105.stdout:2013: left oid 12 (ObjNum 416 snap 49 seq_num 416) 2024-04-20T02:53:24.830 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:24.830+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:24.922 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:24.922 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:24.922 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:24.922 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:24.922 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:24.930 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:24.930+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:24.932 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:25.031 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:25.032+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:25.102 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 185.49790421700001, 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:53:25.103 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:53:25.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:25.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:53:25.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:53:25.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:53:25.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:53:25.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 163.620734777, 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.110 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:25.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:53:25.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:53:25.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 163.260949125, 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.116 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:53:25.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.119 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:25.120 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 163.25058551999999, 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:53:25.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:53:25.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.124 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.960942816, 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:25.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.129 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:25.130 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.95580570600001, 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:25.131 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:53:25.132 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.133 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.134 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.135 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:25.136 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.955532694, 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:25.137 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.138 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.139 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.140 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:53:25.141 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.142 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 144.95550330899999, 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.143 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.144 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:53:25.145 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:53:25.146 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.147 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.26817601, 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.148 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.149 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.150 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.151 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 96.820695787999995, 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:25.152 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.153 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.154 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.155 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 96.803427353999993, 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.156 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.157 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.158 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.159 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 73.630281558999997, 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.160 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.161 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.162 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:53:25.163 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3881 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'232, mlcod=141'228)", 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 40.683159019000001, 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050172267999999999, 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.164 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418659+0000", 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419496+0000", 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00083715799999999996 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.165 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419513+0000", 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7489000000000001e-05 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419524+0000", 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0118e-05 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429881+0000", 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010357762 2024-04-20T02:53:25.166 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429905+0000", 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3546000000000001e-05 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468790+0000", 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038884821 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468832+0000", 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2341e-05 2024-04-20T02:53:25.167 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3898 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'133, mlcod=133'129)", 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 40.631000047000001, 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038834664999999997, 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.168 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470817+0000", 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.169 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471051+0000", 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023384600000000001 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471066+0000", 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4205e-05 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471074+0000", 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4090000000000003e-06 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.170 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472816+0000", 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017419429999999999 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472843+0000", 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7296000000000001e-05 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509592+0000", 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036748496999999998 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.171 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509653+0000", 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1707999999999999e-05 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3899 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'134, mlcod=133'129)", 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 40.613733611000001, 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038218993999999999, 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.172 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488084+0000", 2024-04-20T02:53:25.173 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488262+0000", 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017768600000000001 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488274+0000", 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2388000000000001e-05 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.174 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488283+0000", 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3840000000000003e-06 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489094+0000", 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081045100000000001 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489119+0000", 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4751000000000001e-05 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.175 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526264+0000", 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037144912000000002 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526304+0000", 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0513000000000003e-05 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3891 4.7 e146/145 4:e2c0d65b:::smithi10548823-46:head v 146'105, mlcod=143'100)", 2024-04-20T02:53:25.176 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 40.612753515999998, 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041132308999999999, 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489072+0000", 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9539999999999998e-06 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489082+0000", 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0329000000000001e-05 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489089+0000", 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4959999999999999e-06 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489814+0000", 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072423699999999997 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489837+0000", 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3218999999999999e-05 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530157+0000", 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040319762000000002 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530198+0000", 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0886000000000001e-05 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.180 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.7 146/145 [PushOp(4:e2c0d65b:::smithi10548823-46:43, version: 143'96, data_included: [426944~820638,1468778~227938], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:e2c0d65b:::smithi10548823-46:43@143'96, size: 3311012, copy_subset: [0~3311012], clone_subset: {}, snapset: 45=[]:{3f=[3f,39],43=[41],45=[45]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1696716, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 40.611542163999999, 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039876652999999998, 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490276+0000", 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490785+0000", 2024-04-20T02:53:25.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050941300000000001 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490804+0000", 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9080999999999999e-05 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490823+0000", 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.855e-05 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498138+0000", 2024-04-20T02:53:25.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007315004 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498153+0000", 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5515e-05 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530153+0000", 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031999845999999998 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4016 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'146, mlcod=147'143)", 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.839237918000002, 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054600045999999999, 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262580+0000", 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262733+0000", 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015289700000000001 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262746+0000", 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.375e-05 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262753+0000", 2024-04-20T02:53:25.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6159999999999998e-06 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262817+0000", 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4202999999999995e-05 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262842+0000", 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5296e-05 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317137+0000", 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054294664999999999 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317181+0000", 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4017 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'147, mlcod=147'143)", 2024-04-20T02:53:25.189 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 36.824191231999997, 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039585437000000001, 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277626+0000", 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278058+0000", 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043228299999999999 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278074+0000", 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5775000000000001e-05 2024-04-20T02:53:25.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278083+0000", 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2620000000000004e-06 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286849+0000", 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087652210000000001 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286881+0000", 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1919999999999999e-05 2024-04-20T02:53:25.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317188+0000", 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030307392999999998 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317213+0000", 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4953999999999999e-05 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4340 4.8 e163/114 4:1a8c7fb2:::smithi10548823-47:head v 163'162, mlcod=159'158)", 2024-04-20T02:53:25.193 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 22.189514847000002, 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.04004311, 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:25.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912303+0000", 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912722+0000", 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00041911999999999999 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912738+0000", 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6294e-05 2024-04-20T02:53:25.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912748+0000", 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6059999999999992e-06 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912840+0000", 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2143000000000002e-05 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912867+0000", 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7121e-05 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952298+0000", 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039430999000000001 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952347+0000", 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8990999999999999e-05 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.197 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:25.199 INFO:tasks.rados.rados.0.smithi105.stdout:2008: done (8 left) 2024-04-20T02:53:25.199 INFO:tasks.rados.rados.0.smithi105.stdout:2009: done (7 left) 2024-04-20T02:53:25.199 INFO:tasks.rados.rados.0.smithi105.stdout:2010: done (6 left) 2024-04-20T02:53:25.200 INFO:tasks.rados.rados.0.smithi105.stdout:2011: done (5 left) 2024-04-20T02:53:25.200 INFO:tasks.rados.rados.0.smithi105.stdout:2012: done (4 left) 2024-04-20T02:53:25.200 INFO:tasks.rados.rados.0.smithi105.stdout:2013: done (3 left) 2024-04-20T02:53:25.200 INFO:tasks.rados.rados.0.smithi105.stdout:2014: done (2 left) 2024-04-20T02:53:25.200 INFO:tasks.rados.rados.0.smithi105.stdout:2015: done (1 left) 2024-04-20T02:53:25.200 INFO:tasks.rados.rados.0.smithi105.stdout:2016: done (0 left) 2024-04-20T02:53:25.200 INFO:tasks.rados.rados.0.smithi105.stdout:2017: read oid 43 snap -1 2024-04-20T02:53:25.200 INFO:tasks.rados.rados.0.smithi105.stdout:2017: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:53:25.200 INFO:tasks.rados.rados.0.smithi105.stdout:2018: write (excl) oid 21 current snap is 50 2024-04-20T02:53:25.200 INFO:tasks.rados.rados.0.smithi105.stdout:2018: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-04-20T02:53:25.200 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:25.201 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:53:25.201 INFO:tasks.rados.rados.0.smithi105.stdout:2018: writing smithi10548823-21 from 608519 to 1070414 tid 1 2024-04-20T02:53:25.201 INFO:tasks.rados.rados.0.smithi105.stdout:2018: writing smithi10548823-21 from 1602050 to 2130859 tid 2 2024-04-20T02:53:25.202 INFO:tasks.rados.rados.0.smithi105.stdout:2018: writing smithi10548823-21 from 2585035 to 3008519 tid 3 2024-04-20T02:53:25.202 INFO:tasks.rados.rados.0.smithi105.stdout:2019: tier_promote oid 8 2024-04-20T02:53:25.203 INFO:tasks.rados.rados.0.smithi105.stdout:2017: expect (ObjNum 276 snap 26 seq_num 276) 2024-04-20T02:53:25.203 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:25.199+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:25.300 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:25.300+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:25.332 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:25.332 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:25.332 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:25.332 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:25.342 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:25.372 INFO:tasks.rados.rados.0.smithi105.stdout:2019: got (0) Success 2024-04-20T02:53:25.373 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 136 (ObjNum 336 snap 36 seq_num 336) dirty exists 2024-04-20T02:53:25.373 INFO:tasks.rados.rados.0.smithi105.stdout:2018: finishing write tid 1 to smithi10548823-21 2024-04-20T02:53:25.373 INFO:tasks.rados.rados.0.smithi105.stdout:2018: finishing write tid 2 to smithi10548823-21 2024-04-20T02:53:25.373 INFO:tasks.rados.rados.0.smithi105.stdout:2018: finishing write tid 3 to smithi10548823-21 2024-04-20T02:53:25.373 INFO:tasks.rados.rados.0.smithi105.stdout:2017: done (2 left) 2024-04-20T02:53:25.373 INFO:tasks.rados.rados.0.smithi105.stdout:2019: done (1 left) 2024-04-20T02:53:25.373 INFO:tasks.rados.rados.0.smithi105.stdout:2020: write (excl) oid 48 current snap is 50 2024-04-20T02:53:25.373 INFO:tasks.rados.rados.0.smithi105.stdout:2020: seq_num 418 ranges {625326=537144,1944179=672467} 2024-04-20T02:53:25.378 INFO:tasks.rados.rados.0.smithi105.stdout:2020: writing smithi10548823-48 from 625326 to 1162470 tid 1 2024-04-20T02:53:25.387 INFO:tasks.rados.rados.0.smithi105.stdout:2020: writing smithi10548823-48 from 1944179 to 2616646 tid 2 2024-04-20T02:53:25.387 INFO:tasks.rados.rados.0.smithi105.stdout:2021: read oid 42 snap -1 2024-04-20T02:53:25.387 INFO:tasks.rados.rados.0.smithi105.stdout:2021: expect (ObjNum 375 snap 39 seq_num 375) 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2022: read oid 46 snap 47 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2022: expect (ObjNum 407 snap 46 seq_num 407) 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2023: write (excl) oid 49 current snap is 50 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2018: finishing write tid 4 to smithi10548823-21 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2018: finishing write tid 5 to smithi10548823-21 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2018: oid 21 updating version 0 to 203 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2018: oid 21 updating version 203 to 205 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2018: oid 21 version 205 is already newer than 202 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2018: oid 21 version 205 is already newer than 204 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 21 v 205 (ObjNum 417 snap 50 seq_num 417) dirty exists 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2018: left oid 21 (ObjNum 417 snap 50 seq_num 417) 2024-04-20T02:53:25.388 INFO:tasks.rados.rados.0.smithi105.stdout:2023: seq_num 419 ranges {642133=728746,2119014=532478} 2024-04-20T02:53:25.394 INFO:tasks.rados.rados.0.smithi105.stdout:2023: writing smithi10548823-49 from 642133 to 1370879 tid 1 2024-04-20T02:53:25.401 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:25.400+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:25.402 INFO:tasks.rados.rados.0.smithi105.stdout:2023: writing smithi10548823-49 from 2119014 to 2651492 tid 2 2024-04-20T02:53:25.402 INFO:tasks.rados.rados.0.smithi105.stdout:2018: done (4 left) 2024-04-20T02:53:25.402 INFO:tasks.rados.rados.0.smithi105.stdout:2024: write (excl) oid 1 current snap is 50 2024-04-20T02:53:25.402 INFO:tasks.rados.rados.0.smithi105.stdout:2024: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-04-20T02:53:25.408 INFO:tasks.rados.rados.0.smithi105.stdout:2024: writing smithi10548823-1 from 658940 to 1062935 tid 1 2024-04-20T02:53:25.416 INFO:tasks.rados.rados.0.smithi105.stdout:2024: writing smithi10548823-1 from 1777496 to 2453632 tid 2 2024-04-20T02:53:25.420 INFO:tasks.rados.rados.0.smithi105.stdout:2024: writing smithi10548823-1 from 2950951 to 3058940 tid 3 2024-04-20T02:53:25.420 INFO:tasks.rados.rados.0.smithi105.stdout:2025: set_chunk oid 4 target oid 2024-04-20T02:53:25.441 INFO:tasks.rados.rados.0.smithi105.stdout:2025: set_chunk oid 4 offset: 1810432 length: 3584 target oid ee655843c059ec32bcc89d604b5ab534c586e3361a3100b00a4abb9cdff3ef34 offset: 0 snap -1 2024-04-20T02:53:25.441 INFO:tasks.rados.rados.0.smithi105.stdout:2026: read oid 2 snap -1 2024-04-20T02:53:25.441 INFO:tasks.rados.rados.0.smithi105.stdout:2026: expect (ObjNum 307 snap 33 seq_num 307) 2024-04-20T02:53:25.441 INFO:tasks.rados.rados.0.smithi105.stdout:2027: tier_promote oid 45 2024-04-20T02:53:25.441 INFO:tasks.rados.rados.0.smithi105.stdout:2028: read oid 16 snap -1 2024-04-20T02:53:25.442 INFO:tasks.rados.rados.0.smithi105.stdout:2028: expect deleted 2024-04-20T02:53:25.442 INFO:tasks.rados.rados.0.smithi105.stdout:2029: read oid 27 snap -1 2024-04-20T02:53:25.442 INFO:tasks.rados.rados.0.smithi105.stdout:2029: expect (ObjNum 406 snap 46 seq_num 406) 2024-04-20T02:53:25.442 INFO:tasks.rados.rados.0.smithi105.stdout:2030: write (excl) oid 8 current snap is 50 2024-04-20T02:53:25.442 INFO:tasks.rados.rados.0.smithi105.stdout:2030: seq_num 421 ranges {675747=479244,1835978=536147} 2024-04-20T02:53:25.448 INFO:tasks.rados.rados.0.smithi105.stdout:2030: writing smithi10548823-8 from 675747 to 1154991 tid 1 2024-04-20T02:53:25.455 INFO:tasks.rados.rados.0.smithi105.stdout:2030: writing smithi10548823-8 from 1835978 to 2372125 tid 2 2024-04-20T02:53:25.455 INFO:tasks.rados.rados.0.smithi105.stdout:2031: read oid 21 snap -1 2024-04-20T02:53:25.455 INFO:tasks.rados.rados.0.smithi105.stdout:2031: expect (ObjNum 417 snap 50 seq_num 417) 2024-04-20T02:53:25.455 INFO:tasks.rados.rados.0.smithi105.stdout:2032: read oid 10 snap -1 2024-04-20T02:53:25.455 INFO:tasks.rados.rados.0.smithi105.stdout:2032: expect deleted 2024-04-20T02:53:25.455 INFO:tasks.rados.rados.0.smithi105.stdout:2033: write (excl) oid 32 current snap is 50 2024-04-20T02:53:25.455 INFO:tasks.rados.rados.0.smithi105.stdout:2021: expect (ObjNum 375 snap 39 seq_num 375) 2024-04-20T02:53:25.496 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:25.496 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:25.496 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:25.496 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:25.496 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:25.501 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:25.501+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:25.506 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:25.533 INFO:tasks.rados.rados.0.smithi105.stdout:2022: expect (ObjNum 407 snap 46 seq_num 407) 2024-04-20T02:53:25.602 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:25.602+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:25.677 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:53:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 136.228161231, 2024-04-20T02:53:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:53:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.678 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:25.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:53:25.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:25.682 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 127.17797846400001, 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:53:25.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 127.177913015, 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.687 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:53:25.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:53:25.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 127.177887517, 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:53:25.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2524 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1723287~549075] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 97.390378033999994, 2024-04-20T02:53:25.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054106970999999997, 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2524 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286419+0000", 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286924+0000", 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050494600000000004 2024-04-20T02:53:25.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286944+0000", 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9833000000000001e-05 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286957+0000", 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2901000000000001e-05 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298074+0000", 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011117320999999999 2024-04-20T02:53:25.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298151+0000", 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6909999999999994e-05 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298274+0000", 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012279700000000001 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326267+0000", 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027993284 2024-04-20T02:53:25.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331226+0000", 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049590559999999999 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340442+0000", 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092155740000000007 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340488+0000", 2024-04-20T02:53:25.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6208000000000002e-05 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340528+0000", 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0277000000000001e-05 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3637 4.1 e137/60 4:80c7a726:::smithi10548823-19:head v 137'124, mlcod=120'120)", 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 50.052390475000003, 2024-04-20T02:53:25.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052640605, 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624408+0000", 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624537+0000", 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012880199999999999 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624548+0000", 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0216e-05 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624556+0000", 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6179999999999995e-06 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624603+0000", 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6601999999999999e-05 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624632+0000", 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9277000000000001e-05 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.676999+0000", 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052366639999999999 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.677049+0000", 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0822999999999998e-05 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.706 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3880 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'231, mlcod=141'228)", 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.269895495, 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.061291667000000001, 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406903+0000", 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407359+0000", 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00045623700000000002 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407370+0000", 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1052e-05 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407377+0000", 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1080000000000004e-06 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407429+0000", 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2033999999999997e-05 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407458+0000", 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8050000000000001e-05 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468161+0000", 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060703628000000003 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468196+0000", 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4344999999999997e-05 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3888 4.7 4:e2c0d65b:::smithi10548823-46:head [write 413538~536345,stat] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.251342366999999, 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.059175945000000001, 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3888 2024-04-20T02:53:25.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425456+0000", 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426097+0000", 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00064115099999999996 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426113+0000", 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6118000000000001e-05 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426122+0000", 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.115e-06 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426179+0000", 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6815999999999999e-05 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426316+0000", 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000136606 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426499+0000", 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000183191 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443652+0000", 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017153471999999999 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484036+0000", 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040384097000000001 2024-04-20T02:53:25.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484231+0000", 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000194811 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484363+0000", 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013146300000000001 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484426+0000", 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3563000000000001e-05 2024-04-20T02:53:25.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484633+0000", 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020662299999999999 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3889 4.7 4:e2c0d65b:::smithi10548823-46:head [write 1395221~583781] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.244448392000002, 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.057918795000000002, 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.717 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3889 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432350+0000", 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433490+0000", 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001140606 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433506+0000", 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5828000000000002e-05 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433515+0000", 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8279999999999992e-06 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443692+0000", 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010177222 2024-04-20T02:53:25.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443839+0000", 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014662 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.444072+0000", 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023362200000000001 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484114+0000", 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040041641000000003 2024-04-20T02:53:25.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484244+0000", 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012987200000000001 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484279+0000", 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5301e-05 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490055+0000", 2024-04-20T02:53:25.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057763459999999999 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490115+0000", 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9320999999999997e-05 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490270+0000", 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015498199999999999 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3890 4.7 4:e2c0d65b:::smithi10548823-46:head [write 2556103~658617] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.236434291000002, 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066878325000000002, 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3890 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440363+0000", 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441831+0000", 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001467632 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441842+0000", 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0540000000000001e-05 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441850+0000", 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7260000000000008e-06 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463536+0000", 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02168544 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463681+0000", 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000145086 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463941+0000", 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00025968600000000002 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484317+0000", 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020376452 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490318+0000", 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060012490000000002 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490398+0000", 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9598000000000006e-05 2024-04-20T02:53:25.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506988+0000", 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016589706999999999 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507066+0000", 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8299000000000004e-05 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507243+0000", 2024-04-20T02:53:25.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000177503 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3891 4.7 4:e2c0d65b:::smithi10548823-46:head [setxattr _header (57) in=7b,truncate 3613538] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.234938782, 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088590031, 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.730 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3891 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441859+0000", 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441864+0000", 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8069999999999999e-06 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441869+0000", 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0989999999999998e-06 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441875+0000", 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5820000000000001e-06 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484675+0000", 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042799867999999998 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484817+0000", 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014236899999999999 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485048+0000", 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023057199999999999 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490359+0000", 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053116819999999999 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506963+0000", 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016603562999999998 2024-04-20T02:53:25.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507256+0000", 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00029288699999999998 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530316+0000", 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023060343000000001 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530356+0000", 2024-04-20T02:53:25.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9301999999999997e-05 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530451+0000", 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5037999999999994e-05 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3892 4.7 4:e2c0d65b:::smithi10548823-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.234919093999999, 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088912482000000001, 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.736 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3892 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.737 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441883+0000", 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9579999999999999e-06 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441889+0000", 2024-04-20T02:53:25.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7499999999999997e-06 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441895+0000", 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485431+0000", 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043536321000000003 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485537+0000", 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010612599999999999 2024-04-20T02:53:25.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530489+0000", 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044951996000000001 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530598+0000", 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108937 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530793+0000", 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019441 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.740 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3898 4.8 4:1c68e542:::smithi10548823-8:head [write 447152~403196] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.210576011000001, 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.056917473000000003, 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3898 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.741 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466222+0000", 2024-04-20T02:53:25.742 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466722+0000", 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00049941299999999998 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466734+0000", 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2860999999999999e-05 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466743+0000", 2024-04-20T02:53:25.743 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7159999999999992e-06 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466809+0000", 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6165000000000001e-05 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466928+0000", 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011869199999999999 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:25.744 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.467158+0000", 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000230251 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498338+0000", 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.03117996 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.522958+0000", 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024619318000000001 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.745 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523002+0000", 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4675999999999997e-05 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523082+0000", 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9803999999999997e-05 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523141+0000", 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8767000000000002e-05 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.746 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3899 4.8 4:1c68e542:::smithi10548823-8:head [write 1512185~587450] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.202648244000002, 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060518556000000001, 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3899 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.747 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474149+0000", 2024-04-20T02:53:25.748 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474686+0000", 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053696199999999999 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474703+0000", 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.615e-05 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.749 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474713+0000", 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0829000000000001e-05 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478533+0000", 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003819687 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478640+0000", 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106974 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.750 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478766+0000", 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125859 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523803+0000", 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045036616000000002 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526432+0000", 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026291489999999999 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.751 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534589+0000", 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081572209999999992 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534623+0000", 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534670+0000", 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6968999999999997e-05 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.752 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4016 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 665643~697786] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.415479478999998, 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060676251, 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4016 2024-04-20T02:53:25.753 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.754 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261318+0000", 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262081+0000", 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076285299999999997 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262095+0000", 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3986e-05 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.755 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262118+0000", 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2407000000000001e-05 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262164+0000", 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6248999999999997e-05 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262238+0000", 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4202000000000005e-05 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.756 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262363+0000", 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012437799999999999 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314944+0000", 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052581687000000002 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317407+0000", 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024628300000000001 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.757 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321887+0000", 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0044803020000000002 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321941+0000", 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3769e-05 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321996+0000", 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4988000000000003e-05 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.758 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4017 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 2039745~753122] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.407849681000002, 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053146383999999998, 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4017 2024-04-20T02:53:25.759 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.760 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268948+0000", 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269678+0000", 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072975799999999995 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269693+0000", 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.453e-05 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.761 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269704+0000", 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1623e-05 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277190+0000", 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074859849999999997 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277276+0000", 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5809999999999993e-05 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.762 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277405+0000", 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012914600000000001 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314952+0000", 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037546931999999998 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317441+0000", 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024889840000000001 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.763 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322031+0000", 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004589465 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322059+0000", 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322096+0000", 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591999999999998e-05 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.764 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4074 4.11 4:8b91c759:::smithi10548823-5:head [write 1591248~408851] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.534882291999999, 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053876146, 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4074 2024-04-20T02:53:25.765 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141916+0000", 2024-04-20T02:53:25.766 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142740+0000", 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000823616 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142750+0000", 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6150000000000003e-06 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142756+0000", 2024-04-20T02:53:25.767 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9419999999999998e-06 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2061999999999997e-05 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142815+0000", 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5050000000000004e-06 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.768 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174069+0000", 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031253933999999997 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174193+0000", 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012386 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174438+0000", 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00024561400000000001 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.769 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187472+0000", 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013033375999999999 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.188278+0000", 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00080595500000000002 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195551+0000", 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072733340000000002 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.770 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195631+0000", 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9882999999999994e-05 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195793+0000", 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000162223 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4075 4.11 4:8b91c759:::smithi10548823-5:head [setxattr _header (56) in=7b,truncate 2000099] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:25.771 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.53403136, 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084064655000000002, 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4075 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:25.772 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142767+0000", 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.773 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142778+0000", 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0709e-05 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142783+0000", 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4539999999999998e-06 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142788+0000", 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2850000000000002e-06 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.774 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142845+0000", 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6919000000000002e-05 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142856+0000", 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0936e-05 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186635+0000", 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043778897999999997 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.775 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186756+0000", 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121087 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186967+0000", 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021064599999999999 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.189477+0000", 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025104929999999999 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.776 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.190395+0000", 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00091717399999999998 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226595+0000", 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036200195999999997 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226661+0000", 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6465999999999997e-05 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.777 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226833+0000", 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000171471 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4076 4.11 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 33.533990353999997, 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084363853000000003, 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.778 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4076 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:25.779 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142808+0000", 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142811+0000", 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9229999999999998e-06 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.780 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142818+0000", 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4420000000000003e-06 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142823+0000", 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5199999999999997e-06 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142875+0000", 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1759999999999999e-05 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:25.781 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142880+0000", 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6639999999999997e-06 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187624+0000", 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044743949999999998 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187726+0000", 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0001022 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.782 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226882+0000", 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039155862 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227003+0000", 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120715 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227173+0000", 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017032899999999999 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.783 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4183 3.2 e154/49 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head v 154'234, mlcod=154'232)", 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 32.023861420999999, 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.063121930000000007, 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:25.784 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652936+0000", 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657053+0000", 2024-04-20T02:53:25.785 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041172520000000001 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657067+0000", 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3865e-05 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657076+0000", 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5390000000000007e-06 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:25.786 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657121+0000", 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5114000000000002e-05 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657138+0000", 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7600000000000001e-05 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716021+0000", 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.058882663000000002 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:25.787 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716060+0000", 2024-04-20T02:53:25.788 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8826999999999999e-05 2024-04-20T02:53:25.788 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.788 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.788 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.788 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:25.788 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:25.788 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:25.790 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:25.791 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:25.789+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:25.792 INFO:tasks.rados.rados.0.smithi105.stdout:2020: finishing write tid 1 to smithi10548823-48 2024-04-20T02:53:25.792 INFO:tasks.rados.rados.0.smithi105.stdout:2020: finishing write tid 2 to smithi10548823-48 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2033: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2033: writing smithi10548823-32 from 692554 to 1247047 tid 1 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2033: writing smithi10548823-32 from 1778107 to 2457912 tid 2 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2033: writing smithi10548823-32 from 2889444 to 3092554 tid 3 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2020: finishing write tid 3 to smithi10548823-48 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2020: finishing write tid 4 to smithi10548823-48 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2020: oid 48 updating version 0 to 152 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2020: oid 48 updating version 152 to 153 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2020: oid 48 version 153 is already newer than 151 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 48 v 153 (ObjNum 418 snap 50 seq_num 418) dirty exists 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2020: left oid 48 (ObjNum 418 snap 50 seq_num 418) 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2023: finishing write tid 1 to smithi10548823-49 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2023: finishing write tid 2 to smithi10548823-49 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2023: finishing write tid 3 to smithi10548823-49 2024-04-20T02:53:25.793 INFO:tasks.rados.rados.0.smithi105.stdout:2023: finishing write tid 4 to smithi10548823-49 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2023: oid 49 updating version 0 to 285 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2023: oid 49 version 285 is already newer than 284 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2023: oid 49 updating version 285 to 286 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 49 v 286 (ObjNum 419 snap 50 seq_num 419) dirty exists 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2023: left oid 49 (ObjNum 419 snap 50 seq_num 419) 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2024: finishing write tid 1 to smithi10548823-1 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2024: finishing write tid 2 to smithi10548823-1 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2020: done (13 left) 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2021: done (12 left) 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2022: done (11 left) 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2023: done (10 left) 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2034: set_chunk oid 43 target oid 2024-04-20T02:53:25.794 INFO:tasks.rados.rados.0.smithi105.stdout:2034: set_chunk oid 43 offset: 1038336 length: 1271808 target oid 98d7ee34cb92fe940c36763bc19270c592fe6f0caac861f59a4ee086c8c10df9 offset: 0 snap -1 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2035: read oid 25 snap 48 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2035: expect (ObjNum 383 snap 41 seq_num 383) 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2036: read oid 34 snap -1 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2036: expect (ObjNum 377 snap 40 seq_num 377) 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2037: read oid 50 snap -1 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2037: expect (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2038: read oid 15 snap -1 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2038: expect (ObjNum 400 snap 46 seq_num 400) 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2039: snap_create 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2024: finishing write tid 3 to smithi10548823-1 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2024: finishing write tid 4 to smithi10548823-1 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2024: finishing write tid 5 to smithi10548823-1 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2024: oid 1 updating version 0 to 310 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2024: oid 1 updating version 310 to 311 2024-04-20T02:53:25.795 INFO:tasks.rados.rados.0.smithi105.stdout:2024: oid 1 updating version 311 to 313 2024-04-20T02:53:25.796 INFO:tasks.rados.rados.0.smithi105.stdout:2024: oid 1 version 313 is already newer than 312 2024-04-20T02:53:25.796 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 1 v 313 (ObjNum 420 snap 50 seq_num 420) dirty exists 2024-04-20T02:53:25.796 INFO:tasks.rados.rados.0.smithi105.stdout:2024: left oid 1 (ObjNum 420 snap 50 seq_num 420) 2024-04-20T02:53:25.796 INFO:tasks.rados.rados.0.smithi105.stdout:2025: finishing set_chunk to oid 4 2024-04-20T02:53:25.796 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 4 v 283 (ObjNum 392 snap 44 seq_num 392) dirty exists 2024-04-20T02:53:25.796 INFO:tasks.rados.rados.0.smithi105.stdout:2026: expect (ObjNum 307 snap 33 seq_num 307) 2024-04-20T02:53:25.890 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:25.890+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:25.924 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:25.924 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:25.924 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:25.924 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:25.935 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:25.991 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:25.991+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:26.022 INFO:tasks.rados.rados.0.smithi105.stdout:2027: got (2) No such file or directory 2024-04-20T02:53:26.023 INFO:tasks.rados.rados.0.smithi105.stdout:2027: got expected ENOENT (src dne) 2024-04-20T02:53:26.023 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 286 (ObjNum 129 snap 0 seq_num 59125872) clean dne 2024-04-20T02:53:26.023 INFO:tasks.rados.rados.0.smithi105.stdout:2029: expect (ObjNum 406 snap 46 seq_num 406) 2024-04-20T02:53:26.063 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:26.063 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:26.063 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:26.063 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:26.063 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:26.075 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:26.114 INFO:tasks.rados.rados.0.smithi105.stdout:2031: expect (ObjNum 417 snap 50 seq_num 417) 2024-04-20T02:53:26.115 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:26.114+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:26.193 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:26.193+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:26.246 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:26.246 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:26.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:26.246 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-04-20T02:53:26.246 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:26.255 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:26.294 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:26.294+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:26.326 INFO:tasks.rados.rados.0.smithi105.stdout:2030: finishing write tid 1 to smithi10548823-8 2024-04-20T02:53:26.326 INFO:tasks.rados.rados.0.smithi105.stdout:2030: finishing write tid 2 to smithi10548823-8 2024-04-20T02:53:26.326 INFO:tasks.rados.rados.0.smithi105.stdout:2030: finishing write tid 3 to smithi10548823-8 2024-04-20T02:53:26.326 INFO:tasks.rados.rados.0.smithi105.stdout:2030: finishing write tid 4 to smithi10548823-8 2024-04-20T02:53:26.326 INFO:tasks.rados.rados.0.smithi105.stdout:2030: oid 8 updating version 0 to 173 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2030: oid 8 updating version 173 to 174 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2030: oid 8 version 174 is already newer than 172 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 8 v 174 (ObjNum 421 snap 50 seq_num 421) dirty exists 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2030: left oid 8 (ObjNum 421 snap 50 seq_num 421) 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2033: finishing write tid 1 to smithi10548823-32 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2033: finishing write tid 2 to smithi10548823-32 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2033: finishing write tid 3 to smithi10548823-32 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2024: done (15 left) 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2025: done (14 left) 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2026: done (13 left) 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2027: done (12 left) 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2028: done (11 left) 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2029: done (10 left) 2024-04-20T02:53:26.327 INFO:tasks.rados.rados.0.smithi105.stdout:2030: done (9 left) 2024-04-20T02:53:26.328 INFO:tasks.rados.rados.0.smithi105.stdout:2031: done (8 left) 2024-04-20T02:53:26.328 INFO:tasks.rados.rados.0.smithi105.stdout:2032: done (7 left) 2024-04-20T02:53:26.328 INFO:tasks.rados.rados.0.smithi105.stdout:2039: done (6 left) 2024-04-20T02:53:26.328 INFO:tasks.rados.rados.0.smithi105.stdout:2040: read oid 48 snap -1 2024-04-20T02:53:26.328 INFO:tasks.rados.rados.0.smithi105.stdout:2040: expect (ObjNum 418 snap 50 seq_num 418) 2024-04-20T02:53:26.328 INFO:tasks.rados.rados.0.smithi105.stdout:2041: read oid 24 snap 49 2024-04-20T02:53:26.328 INFO:tasks.rados.rados.0.smithi105.stdout:2041: expect deleted 2024-04-20T02:53:26.328 INFO:tasks.rados.rados.0.smithi105.stdout:2042: set_chunk oid 2 target oid 2024-04-20T02:53:26.333 INFO:tasks.rados.rados.0.smithi105.stdout:2042: set_chunk oid 2 offset: 1509888 length: 231936 target oid 87d5a46c4cd24522f3189c2ab55cdb76546235a0682541882e7374811adfbc21 offset: 0 snap -1 2024-04-20T02:53:26.333 INFO:tasks.rados.rados.0.smithi105.stdout:2043: tier_promote oid 9 2024-04-20T02:53:26.333 INFO:tasks.rados.rados.0.smithi105.stdout:2033: finishing write tid 4 to smithi10548823-32 2024-04-20T02:53:26.333 INFO:tasks.rados.rados.0.smithi105.stdout:2033: finishing write tid 5 to smithi10548823-32 2024-04-20T02:53:26.334 INFO:tasks.rados.rados.0.smithi105.stdout:2033: oid 32 updating version 0 to 103 2024-04-20T02:53:26.334 INFO:tasks.rados.rados.0.smithi105.stdout:2033: oid 32 updating version 103 to 104 2024-04-20T02:53:26.334 INFO:tasks.rados.rados.0.smithi105.stdout:2033: oid 32 updating version 104 to 105 2024-04-20T02:53:26.334 INFO:tasks.rados.rados.0.smithi105.stdout:2033: oid 32 version 105 is already newer than 102 2024-04-20T02:53:26.334 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 32 v 105 (ObjNum 422 snap 50 seq_num 422) dirty exists 2024-04-20T02:53:26.334 INFO:tasks.rados.rados.0.smithi105.stdout:2033: left oid 32 (ObjNum 422 snap 50 seq_num 422) 2024-04-20T02:53:26.334 INFO:tasks.rados.rados.0.smithi105.stdout:2034: finishing set_chunk to oid 43 2024-04-20T02:53:26.334 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 43 set_chunk returned error code -95 offset: 1038336 length: 1271808 2024-04-20T02:53:26.334 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 43 v 0 (ObjNum 276 snap 26 seq_num 276) dirty exists 2024-04-20T02:53:26.334 INFO:tasks.rados.rados.0.smithi105.stdout:2035: expect (ObjNum 383 snap 41 seq_num 383) 2024-04-20T02:53:26.389 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:26.390 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:26.390 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:26.390 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:26.394 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:26.394+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:26.400 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:26.495 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:26.495+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:26.544 INFO:tasks.rados.rados.0.smithi105.stdout:2038: expect (ObjNum 400 snap 46 seq_num 400) 2024-04-20T02:53:26.568 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:26.568 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:26.569 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:26.569 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:26.579 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:26.595 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:26.596+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:26.632 INFO:tasks.rados.rados.0.smithi105.stdout:2036: expect (ObjNum 377 snap 40 seq_num 377) 2024-04-20T02:53:26.690 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:53:26.696 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:26.696+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:26.707 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:26.707 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:26.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:26.707 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [] 2024-04-20T02:53:26.707 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:26.717 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:26.789 INFO:tasks.rados.rados.0.smithi105.stdout:2037: expect (ObjNum 350 snap 37 seq_num 350) 2024-04-20T02:53:26.797 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:26.797+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:26.842 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:26.859 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:26.874 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:26.874 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:26.889 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:26.892 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:26.913 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:26.914 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:26.914+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:26.946 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:26.965 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:26.985 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:26.998 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:26.998+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:27.004 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:27.006 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:27.006 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:27.006 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:27.006 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:27.006 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:27.006 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:27.006 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:27.006 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:27.039 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:27.039 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:27.050 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:27.088 INFO:tasks.rados.rados.0.smithi105.stdout:2040: expect (ObjNum 418 snap 50 seq_num 418) 2024-04-20T02:53:27.099 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:27.099+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:27.200 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:27.200+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:27.203 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:53:27.223 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:27.223 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:27.234 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:27.280 INFO:tasks.rados.rados.0.smithi105.stdout:2042: finishing set_chunk to oid 2 2024-04-20T02:53:27.281 INFO:tasks.rados.rados.0.smithi105.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 1509888 length: 231936 2024-04-20T02:53:27.281 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 2 v 0 (ObjNum 307 snap 33 seq_num 307) dirty exists 2024-04-20T02:53:27.281 INFO:tasks.rados.rados.0.smithi105.stdout:2043: got (2) No such file or directory 2024-04-20T02:53:27.281 INFO:tasks.rados.rados.0.smithi105.stdout:2043: got expected ENOENT (src dne) 2024-04-20T02:53:27.281 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 9 v 313 (ObjNum 2081 snap 0 seq_num 188080400) dirty dne 2024-04-20T02:53:27.281 INFO:tasks.rados.rados.0.smithi105.stdout:2033: done (9 left) 2024-04-20T02:53:27.281 INFO:tasks.rados.rados.0.smithi105.stdout:2034: done (8 left) 2024-04-20T02:53:27.281 INFO:tasks.rados.rados.0.smithi105.stdout:2035: done (7 left) 2024-04-20T02:53:27.281 INFO:tasks.rados.rados.0.smithi105.stdout:2036: done (6 left) 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2037: done (5 left) 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2038: done (4 left) 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2040: done (3 left) 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2041: done (2 left) 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2042: done (1 left) 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2043: done (0 left) 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2044: read oid 49 snap -1 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2044: expect (ObjNum 419 snap 50 seq_num 419) 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2045: read oid 11 snap -1 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2045: expect deleted 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2046: tier_promote oid 45 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2047: read oid 40 snap -1 2024-04-20T02:53:27.282 INFO:tasks.rados.rados.0.smithi105.stdout:2047: expect (ObjNum 386 snap 41 seq_num 386) 2024-04-20T02:53:27.283 INFO:tasks.rados.rados.0.smithi105.stdout:2048: write oid 24 current snap is 51 2024-04-20T02:53:27.283 INFO:tasks.rados.rados.0.smithi105.stdout:2048: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-04-20T02:53:27.287 INFO:tasks.rados.rados.0.smithi105.stdout:2048: writing smithi10548823-24 from 709361 to 1339103 tid 1 2024-04-20T02:53:27.294 INFO:tasks.rados.rados.0.smithi105.stdout:2048: writing smithi10548823-24 from 1836589 to 2376405 tid 2 2024-04-20T02:53:27.298 INFO:tasks.rados.rados.0.smithi105.stdout:2048: writing smithi10548823-24 from 3033220 to 3109361 tid 3 2024-04-20T02:53:27.298 INFO:tasks.rados.rados.0.smithi105.stdout:2049: read oid 32 snap 49 2024-04-20T02:53:27.298 INFO:tasks.rados.rados.0.smithi105.stdout:2049: expect deleted 2024-04-20T02:53:27.306 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 6 2024-04-20T02:53:27.306 INFO:tasks.rados.rados.0.smithi105.stdout:2044: expect (ObjNum 419 snap 50 seq_num 419) 2024-04-20T02:53:27.306 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:27.306+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:27.402 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:27.402+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:27.416 INFO:tasks.rados.rados.0.smithi105.stdout:2047: expect (ObjNum 386 snap 41 seq_num 386) 2024-04-20T02:53:27.428 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:27.428 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:27.439 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:27.502 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:27.502+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:27.603 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:27.603+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:27.619 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:27.623 INFO:tasks.rados.rados.0.smithi105.stdout:2046: got (2) No such file or directory 2024-04-20T02:53:27.623 INFO:tasks.rados.rados.0.smithi105.stdout:2046: got expected ENOENT (src dne) 2024-04-20T02:53:27.623 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 45 v 286 (ObjNum 129 snap 0 seq_num 59125872) clean dne 2024-04-20T02:53:27.623 INFO:tasks.rados.rados.0.smithi105.stdout:2044: done (5 left) 2024-04-20T02:53:27.623 INFO:tasks.rados.rados.0.smithi105.stdout:2045: done (4 left) 2024-04-20T02:53:27.623 INFO:tasks.rados.rados.0.smithi105.stdout:2046: done (3 left) 2024-04-20T02:53:27.623 INFO:tasks.rados.rados.0.smithi105.stdout:2047: done (2 left) 2024-04-20T02:53:27.623 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 2 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2049: done (1 left) 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout: waiting on 1 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: finishing write tid 1 to smithi10548823-24 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: finishing write tid 2 to smithi10548823-24 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: finishing write tid 3 to smithi10548823-24 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: finishing write tid 4 to smithi10548823-24 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: finishing write tid 5 to smithi10548823-24 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: oid 24 updating version 0 to 502 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: oid 24 version 502 is already newer than 501 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: oid 24 version 502 is already newer than 500 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: oid 24 updating version 502 to 503 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:update_object_version oid 24 v 503 (ObjNum 423 snap 51 seq_num 423) dirty exists 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: left oid 24 (ObjNum 423 snap 51 seq_num 423) 2024-04-20T02:53:27.624 INFO:tasks.rados.rados.0.smithi105.stdout:2048: done (0 left) 2024-04-20T02:53:27.704 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:27.704+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:27.752 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:27.753 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:27.763 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:27.805 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:27.805+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:27.906 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:27.906+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:27.931 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:27.931 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:27.941 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:28.007 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:28.007+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:28.069 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:28.069 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:28.080 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:28.108 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:28.108+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:28.152 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:53:28.209 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:28.209+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:28.213 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:28.213 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:28.224 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:28.310 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:28.310+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:28.366 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:28.366 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:28.376 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:28.417 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:28.417+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:28.512 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:28.512+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:28.545 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:28.545 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:28.555 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:28.612 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:28.613+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:28.683 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:28.683 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:28.693 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:28.714 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:28.714+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:28.815 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:28.815+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:28.822 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:28.822 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:28.832 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:28.916 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:28.916+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:28.960 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:28.960 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:28.971 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:29.016 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:29.017+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:29.018 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:29.098 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:29.098 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:29.108 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:29.117 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:29.117+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:29.175 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 3, 6, 7, 5, 2, 4] 2024-04-20T02:53:29.176 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:53:29.176 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-20T02:53:29.176 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-20T02:53:29.177 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:29.198 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:29.218 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:29.218+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:29.218 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:29.238 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:29.239 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:29.240 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:29.250 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:29.257 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:29.277 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:29.297 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:29.319 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:29.320+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:29.380 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:29.380 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:29.380 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:29.380 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:29.380 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:29.390 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:29.431 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:29.432+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:29.511 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:53:29.511 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:29.511 INFO:teuthology.orchestra.run.smithi105.stdout: "epoch": 190, 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "fsid": "a524a365-eaec-4ebd-aa4d-0d76d55a252f", 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "created": "2024-04-20T02:50:07.356902+0000", 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "modified": "2024-04-20T02:53:29.144998+0000", 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "last_up_change": "2024-04-20T02:53:06.173366+0000", 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "last_in_change": "2024-04-20T02:52:50.391054+0000", 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_num": 5804032, 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_set": [ 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "nodeep-scrub", 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "pglog_hardlimit", 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "purged_snapdirs", 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_deletes", 2024-04-20T02:53:29.512 INFO:teuthology.orchestra.run.smithi105.stdout: "sortbitwise" 2024-04-20T02:53:29.513 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:29.513 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_version": 7, 2024-04-20T02:53:29.513 INFO:teuthology.orchestra.run.smithi105.stdout: "full_ratio": 0.94999998807907104, 2024-04-20T02:53:29.513 INFO:teuthology.orchestra.run.smithi105.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-20T02:53:29.513 INFO:teuthology.orchestra.run.smithi105.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-20T02:53:29.513 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_snapshot": "", 2024-04-20T02:53:29.513 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_max": 4, 2024-04-20T02:53:29.513 INFO:teuthology.orchestra.run.smithi105.stdout: "max_osd": 8, 2024-04-20T02:53:29.513 INFO:teuthology.orchestra.run.smithi105.stdout: "require_min_compat_client": "luminous", 2024-04-20T02:53:29.514 INFO:teuthology.orchestra.run.smithi105.stdout: "min_compat_client": "luminous", 2024-04-20T02:53:29.514 INFO:teuthology.orchestra.run.smithi105.stdout: "require_osd_release": "reef", 2024-04-20T02:53:29.514 INFO:teuthology.orchestra.run.smithi105.stdout: "allow_crimson": false, 2024-04-20T02:53:29.514 INFO:teuthology.orchestra.run.smithi105.stdout: "pools": [ 2024-04-20T02:53:29.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.514 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 1, 2024-04-20T02:53:29.514 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": ".mgr", 2024-04-20T02:53:29.514 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:17.498569+0000", 2024-04-20T02:53:29.515 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-20T02:53:29.515 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool", 2024-04-20T02:53:29.515 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:53:29.516 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:53:29.516 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:53:29.516 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:53:29.516 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:53:29.516 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:53:29.516 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:53:29.516 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:53:29.516 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:53:29.516 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:53:29.516 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 1, 2024-04-20T02:53:29.517 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 1, 2024-04-20T02:53:29.517 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 1, 2024-04-20T02:53:29.517 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 1, 2024-04-20T02:53:29.517 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 1, 2024-04-20T02:53:29.517 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:53:29.517 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "0.0", 2024-04-20T02:53:29.517 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 0, 2024-04-20T02:53:29.517 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 0, 2024-04-20T02:53:29.517 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 0, 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "0'0", 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "0'0" 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "17", 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 0, 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 0, 2024-04-20T02:53:29.518 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:53:29.519 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:53:29.519 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:53:29.519 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:53:29.519 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:53:29.519 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:53:29.519 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:53:29.519 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:53:29.519 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:53:29.519 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:53:29.519 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:53:29.520 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "options": { 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_max": 32, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_min": 1 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": { 2024-04-20T02:53:29.521 INFO:teuthology.orchestra.run.smithi105.stdout: "mgr": {} 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 7.8899998664855957, 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 7.8899998664855957, 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 0.37999999523162842, 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 3, 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 3, 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.522 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 2, 2024-04-20T02:53:29.523 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": "rbd", 2024-04-20T02:53:29.523 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:25.955448+0000", 2024-04-20T02:53:29.523 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-20T02:53:29.523 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool", 2024-04-20T02:53:29.523 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:53:29.523 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:53:29.523 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:53:29.523 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:53:29.524 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:53:29.524 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:53:29.524 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:53:29.524 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:53:29.524 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:53:29.524 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:53:29.524 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 8, 2024-04-20T02:53:29.524 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 8, 2024-04-20T02:53:29.524 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 8, 2024-04-20T02:53:29.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 8, 2024-04-20T02:53:29.525 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 8, 2024-04-20T02:53:29.525 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:53:29.525 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "0.0", 2024-04-20T02:53:29.525 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 0, 2024-04-20T02:53:29.525 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 0, 2024-04-20T02:53:29.525 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 0, 2024-04-20T02:53:29.525 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "0'0", 2024-04-20T02:53:29.525 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "0'0" 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "20", 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 0, 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 0, 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:53:29.526 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:53:29.527 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:53:29.527 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:53:29.527 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:53:29.527 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:53:29.527 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:53:29.527 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:53:29.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:53:29.528 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:53:29.528 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:53:29.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:53:29.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:53:29.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:53:29.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:53:29.528 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:53:29.528 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:53:29.529 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:53:29.529 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:53:29.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.529 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:53:29.529 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:53:29.529 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:53:29.530 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:53:29.530 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:53:29.530 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:53:29.530 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:53:29.530 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:53:29.530 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:53:29.530 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:53:29.530 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:53:29.531 INFO:teuthology.orchestra.run.smithi105.stdout: "options": {}, 2024-04-20T02:53:29.531 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": { 2024-04-20T02:53:29.531 INFO:teuthology.orchestra.run.smithi105.stdout: "rbd": {} 2024-04-20T02:53:29.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.531 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:53:29.531 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 2, 2024-04-20T02:53:29.531 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 2, 2024-04-20T02:53:29.531 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 0.75, 2024-04-20T02:53:29.531 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 1.5, 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 1.5, 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 3, 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": "low_tier", 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:33.168857+0000", 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 1, 2024-04-20T02:53:29.532 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool", 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 4, 2024-04-20T02:53:29.533 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 4, 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 4, 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 4, 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 4, 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "0.0", 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 0, 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 0, 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 0, 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "0'0", 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "0'0" 2024-04-20T02:53:29.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "23", 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "0", 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 0, 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 0, 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:53:29.535 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:53:29.536 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:53:29.537 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:53:29.537 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:53:29.537 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:53:29.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.537 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:53:29.537 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:53:29.537 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:53:29.537 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:53:29.537 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:53:29.537 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "options": {}, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": {}, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 3.9700000286102295, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 3.9700000286102295, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 0.62999999523162842, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 2.5, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 2.5, 2024-04-20T02:53:29.538 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 4, 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_name": "unique_pool_0", 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "create_time": "2024-04-20T02:50:33.714259+0000", 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "flags": 8193, 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "type": 1, 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 3, 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "min_size": 2, 2024-04-20T02:53:29.539 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_rule": 0, 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_count": 0, 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_target": 0, 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_barrier": 0, 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "object_hash": 2, 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_autoscale_mode": "off", 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num": 17, 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num": 16, 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_placement_num_target": 16, 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_target": 16, 2024-04-20T02:53:29.540 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_num_pending": 17, 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_pg_merge_meta": { 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "source_pgid": "4.11", 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "ready_epoch": 189, 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_started": 188, 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_epoch_clean": 188, 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "source_version": "183'233", 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "target_version": "184'139" 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_change": "189", 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend": "0", 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_prenautilus": "189", 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "last_force_op_resend_preluminous": "187", 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "auid": 0, 2024-04-20T02:53:29.541 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_mode": "selfmanaged", 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_seq": 99, 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "snap_epoch": 187, 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "pool_snaps": [], 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps": "[]", 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_bytes": 0, 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "quota_max_objects": 0, 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "tiers": [], 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "tier_of": -1, 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "read_tier": -1, 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "write_tier": -1, 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_mode": "none", 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_bytes": 0, 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "target_max_objects": 0, 2024-04-20T02:53:29.542 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_flush_age": 0, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "cache_min_evict_age": 0, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profile": "", 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_params": { 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "none" 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_period": 0, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_count": 0, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "use_gmt_hitset": true, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "min_read_recency_for_promote": 0, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "min_write_recency_for_promote": 0, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_grade_decay_rate": 0, 2024-04-20T02:53:29.543 INFO:teuthology.orchestra.run.smithi105.stdout: "hit_set_search_last_n": 0, 2024-04-20T02:53:29.544 INFO:teuthology.orchestra.run.smithi105.stdout: "grade_table": [], 2024-04-20T02:53:29.544 INFO:teuthology.orchestra.run.smithi105.stdout: "stripe_width": 0, 2024-04-20T02:53:29.544 INFO:teuthology.orchestra.run.smithi105.stdout: "expected_num_objects": 0, 2024-04-20T02:53:29.544 INFO:teuthology.orchestra.run.smithi105.stdout: "fast_read": false, 2024-04-20T02:53:29.544 INFO:teuthology.orchestra.run.smithi105.stdout: "options": { 2024-04-20T02:53:29.544 INFO:teuthology.orchestra.run.smithi105.stdout: "dedup_cdc_chunk_size": 131072, 2024-04-20T02:53:29.544 INFO:teuthology.orchestra.run.smithi105.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-04-20T02:53:29.544 INFO:teuthology.orchestra.run.smithi105.stdout: "dedup_tier": 3, 2024-04-20T02:53:29.544 INFO:teuthology.orchestra.run.smithi105.stdout: "fingerprint_algorithm": "sha256" 2024-04-20T02:53:29.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "application_metadata": { 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "rados": {} 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "read_balance": { 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "score_acting": 2.3499999046325684, 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "score_stable": 2.3499999046325684, 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "optimal_score": 0.75, 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_acting": 1.7599999904632568, 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "raw_score_stable": 1.7599999904632568, 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity_weighted": 0.99730110168457031, 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity": 1, 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: "average_primary_affinity_weighted": 1 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.545 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "osds": [ 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 0, 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6", 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 0.8316650390625, 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 15, 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 181, 2024-04-20T02:53:29.546 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6812", 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.548 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6813", 2024-04-20T02:53:29.548 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:53:29.548 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.548 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.548 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.548 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:29.548 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.548 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.548 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.549 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6815", 2024-04-20T02:53:29.549 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:53:29.549 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.549 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.549 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:29.550 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.550 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.550 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6814", 2024-04-20T02:53:29.550 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43330 2024-04-20T02:53:29.550 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.550 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.551 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6812/43330", 2024-04-20T02:53:29.551 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6813/43330", 2024-04-20T02:53:29.551 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6815/43330", 2024-04-20T02:53:29.551 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6814/43330", 2024-04-20T02:53:29.551 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:29.551 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:29.552 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:29.552 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.552 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 1, 2024-04-20T02:53:29.552 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "c5ae8a64-fc7c-47ab-af6a-4b762a840645", 2024-04-20T02:53:29.553 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:29.553 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:29.553 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 0.8221893310546875, 2024-04-20T02:53:29.553 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:29.553 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:53:29.553 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:53:29.553 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 13, 2024-04-20T02:53:29.554 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 177, 2024-04-20T02:53:29.554 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:53:29.554 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:29.554 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:29.554 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.555 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.555 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6804", 2024-04-20T02:53:29.555 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:53:29.555 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.555 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.556 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:29.556 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.556 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.556 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.556 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6805", 2024-04-20T02:53:29.556 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:53:29.556 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.557 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.557 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:29.557 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.557 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.558 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6807", 2024-04-20T02:53:29.558 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:53:29.558 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.558 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.558 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:29.559 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.559 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.559 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6806", 2024-04-20T02:53:29.559 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43331 2024-04-20T02:53:29.559 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.560 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.560 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6804/43331", 2024-04-20T02:53:29.560 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6805/43331", 2024-04-20T02:53:29.560 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6807/43331", 2024-04-20T02:53:29.560 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6806/43331", 2024-04-20T02:53:29.561 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:29.561 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:29.561 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:29.561 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.561 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 2, 2024-04-20T02:53:29.562 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "336e852e-5fa6-4ac0-b053-d31ad40d7c4e", 2024-04-20T02:53:29.562 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:29.562 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 0, 2024-04-20T02:53:29.562 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 0, 2024-04-20T02:53:29.562 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:29.562 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 13, 2024-04-20T02:53:29.563 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 121, 2024-04-20T02:53:29.563 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 138, 2024-04-20T02:53:29.563 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 27, 2024-04-20T02:53:29.563 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 122, 2024-04-20T02:53:29.563 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:29.563 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:29.563 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.564 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.564 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6800", 2024-04-20T02:53:29.564 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66776 2024-04-20T02:53:29.564 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.564 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6801", 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66776 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.565 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6803", 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66776 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6802", 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66776 2024-04-20T02:53:29.566 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6800/66776", 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6801/66776", 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6803/66776", 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6802/66776", 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 3, 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "695cab3a-74d8-4046-ac61-2656dc0d893e", 2024-04-20T02:53:29.567 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 13, 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 176, 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6808", 2024-04-20T02:53:29.568 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6809", 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6811", 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.105:6810", 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.105:6808/43329", 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.105:6809/43329", 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.105:6811/43329", 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.105:6810/43329", 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 4, 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "1d41da44-d894-40ac-acaa-aa0df15296c6", 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 0, 2024-04-20T02:53:29.571 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 0, 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 0.9784088134765625, 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 15, 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 149, 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 167, 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 145, 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 150, 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6808", 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66272 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.572 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6809", 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66272 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6811", 2024-04-20T02:53:29.573 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66272 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6810", 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 66272 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6808/66272", 2024-04-20T02:53:29.574 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6809/66272", 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6811/66272", 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6810/66272", 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 5, 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "3e1c33fa-6c8c-4898-99bc-462ecef25f30", 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 15, 2024-04-20T02:53:29.575 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 25, 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 38, 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 189, 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 26, 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6802", 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.576 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6804", 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6807", 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:53:29.577 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6806", 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 52590 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6802/52590", 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6804/52590", 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6807/52590", 2024-04-20T02:53:29.578 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6806/52590", 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 6, 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "fd1fc124-7cf8-4628-9076-1f27d6385676", 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:53:29.579 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 13, 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 178, 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6800", 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6801", 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6805", 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6803", 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43329 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6800/43329", 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6801/43329", 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6805/43329", 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6803/43329", 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:29.582 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 7, 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "uuid": "53ae8193-5c3e-4698-aacd-df2ad1db9143", 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "up": 1, 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "in": 1, 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "weight": 1, 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_affinity": 1, 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_begin": 0, 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "last_clean_end": 0, 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "up_from": 15, 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "up_thru": 154, 2024-04-20T02:53:29.583 INFO:teuthology.orchestra.run.smithi105.stdout: "down_at": 0, 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: "lost_at": 0, 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addrs": { 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6812", 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addrs": { 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.584 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6813", 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addrs": { 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6815", 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addrs": { 2024-04-20T02:53:29.585 INFO:teuthology.orchestra.run.smithi105.stdout: "addrvec": [ 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: "type": "v2", 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: "addr": "172.21.15.138:6814", 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: "nonce": 43351 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: "public_addr": "172.21.15.138:6812/43351", 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: "cluster_addr": "172.21.15.138:6813/43351", 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_back_addr": "172.21.15.138:6815/43351", 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: "heartbeat_front_addr": "172.21.15.138:6814/43351", 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: "state": [ 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: "exists", 2024-04-20T02:53:29.586 INFO:teuthology.orchestra.run.smithi105.stdout: "up" 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_xinfo": [ 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 0, 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:44.296659+0000", 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.587 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 1, 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:23.214623+0000", 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 2, 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:29.588 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 65536, 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:56.806719+0000", 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 122 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 3, 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:58.973108+0000", 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.589 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 4, 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 49625, 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:38.383284+0000", 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 150 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 5, 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:29.590 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:29.483129+0000", 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 26 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 6, 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:18.346058+0000", 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.591 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "osd": 7, 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "down_stamp": "0.000000", 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_probability": 0, 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "laggy_interval": 0, 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "features": 4540138322906710015, 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "old_weight": 0, 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "last_purged_snaps_scrub": "2024-04-20T02:52:29.823916+0000", 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "dead_epoch": 0 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_upmap": [ 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "pgid": "2.0", 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: "osds": [ 2024-04-20T02:53:29.592 INFO:teuthology.orchestra.run.smithi105.stdout: 6, 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: 0, 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: 2 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_upmap_items": [ 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: "pgid": "2.0", 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: "mappings": [ 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: "from": 0, 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: "to": 7 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.593 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_upmap_primaries": [], 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "pg_temp": [], 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "primary_temp": [], 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "blocklist": {}, 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "range_blocklist": {}, 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "erasure_code_profiles": { 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "default": { 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "crush-failure-domain": "osd", 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "k": "2", 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "m": "1", 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "plugin": "jerasure", 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: "technique": "reed_sol_van" 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.594 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: "removed_snaps_queue": [ 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: "pool": 4, 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: "snaps": [ 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 63, 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 65, 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.595 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 69, 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 71, 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 75, 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 77, 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 2 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.596 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 80, 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 83, 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 3 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 90, 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 1 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: "begin": 92, 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: "length": 2 2024-04-20T02:53:29.597 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: ], 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: "new_removed_snaps": [], 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: "new_purged_snaps": [], 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: "crush_node_flags": {}, 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: "device_class_flags": {}, 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_mode": { 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_mode_enabled": false, 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_bucket_count": 0, 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: "degraded_stretch_mode": 0, 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: "recovering_stretch_mode": 0, 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: "stretch_mode_bucket": 0 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:29.598 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:29.599 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:29.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:29.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:29.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:29.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:29.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:29.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:29.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:29.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:29.601 INFO:tasks.thrashosds.thrasher:j is {'epoch': 190, 'fsid': 'a524a365-eaec-4ebd-aa4d-0d76d55a252f', 'created': '2024-04-20T02:50:07.356902+0000', 'modified': '2024-04-20T02:53:29.144998+0000', 'last_up_change': '2024-04-20T02:53:06.173366+0000', 'last_in_change': '2024-04-20T02:52:50.391054+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-20T02:50:17.498569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-20T02:50:25.955448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-04-20T02:50:33.168857+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_acting': 3.9700000286102295, 'score_stable': 3.9700000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-20T02:50:33.714259+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 17, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 17, 'last_pg_merge_meta': {'source_pgid': '4.11', 'ready_epoch': 189, 'last_epoch_started': 188, 'last_epoch_clean': 188, 'source_version': "183'233", 'target_version': "184'139"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '189', 'last_force_op_resend_preluminous': '187', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 99, 'snap_epoch': 187, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_acting': 2.3499999046325684, 'score_stable': 2.3499999046325684, 'optimal_score': 0.75, 'raw_score_acting': 1.7599999904632568, 'raw_score_stable': 1.7599999904632568, 'primary_affinity_weighted': 0.9973011016845703, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6', 'up': 1, 'in': 1, 'weight': 0.8316650390625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 181, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6812', 'nonce': 43330}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6813', 'nonce': 43330}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6815', 'nonce': 43330}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6814', 'nonce': 43330}]}, 'public_addr': '172.21.15.105:6812/43330', 'cluster_addr': '172.21.15.105:6813/43330', 'heartbeat_back_addr': '172.21.15.105:6815/43330', 'heartbeat_front_addr': '172.21.15.105:6814/43330', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c5ae8a64-fc7c-47ab-af6a-4b762a840645', 'up': 1, 'in': 1, 'weight': 0.8221893310546875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 177, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6804', 'nonce': 43331}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6805', 'nonce': 43331}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6807', 'nonce': 43331}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6806', 'nonce': 43331}]}, 'public_addr': '172.21.15.105:6804/43331', 'cluster_addr': '172.21.15.105:6805/43331', 'heartbeat_back_addr': '172.21.15.105:6807/43331', 'heartbeat_front_addr': '172.21.15.105:6806/43331', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '336e852e-5fa6-4ac0-b053-d31ad40d7c4e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 121, 'up_from': 138, 'up_thru': 27, 'down_at': 122, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6800', 'nonce': 66776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6801', 'nonce': 66776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6803', 'nonce': 66776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6802', 'nonce': 66776}]}, 'public_addr': '172.21.15.105:6800/66776', 'cluster_addr': '172.21.15.105:6801/66776', 'heartbeat_back_addr': '172.21.15.105:6803/66776', 'heartbeat_front_addr': '172.21.15.105:6802/66776', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '695cab3a-74d8-4046-ac61-2656dc0d893e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 176, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6808', 'nonce': 43329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6809', 'nonce': 43329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6811', 'nonce': 43329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6810', 'nonce': 43329}]}, 'public_addr': '172.21.15.105:6808/43329', 'cluster_addr': '172.21.15.105:6809/43329', 'heartbeat_back_addr': '172.21.15.105:6811/43329', 'heartbeat_front_addr': '172.21.15.105:6810/43329', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '1d41da44-d894-40ac-acaa-aa0df15296c6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.9784088134765625, 'last_clean_begin': 15, 'last_clean_end': 149, 'up_from': 167, 'up_thru': 145, 'down_at': 150, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6808', 'nonce': 66272}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6809', 'nonce': 66272}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6811', 'nonce': 66272}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6810', 'nonce': 66272}]}, 'public_addr': '172.21.15.138:6808/66272', 'cluster_addr': '172.21.15.138:6809/66272', 'heartbeat_back_addr': '172.21.15.138:6811/66272', 'heartbeat_front_addr': '172.21.15.138:6810/66272', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '3e1c33fa-6c8c-4898-99bc-462ecef25f30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 15, 'last_clean_end': 25, 'up_from': 38, 'up_thru': 189, 'down_at': 26, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6802', 'nonce': 52590}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6804', 'nonce': 52590}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6807', 'nonce': 52590}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6806', 'nonce': 52590}]}, 'public_addr': '172.21.15.138:6802/52590', 'cluster_addr': '172.21.15.138:6804/52590', 'heartbeat_back_addr': '172.21.15.138:6807/52590', 'heartbeat_front_addr': '172.21.15.138:6806/52590', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fd1fc124-7cf8-4628-9076-1f27d6385676', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 178, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6800', 'nonce': 43329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6801', 'nonce': 43329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6805', 'nonce': 43329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6803', 'nonce': 43329}]}, 'public_addr': '172.21.15.138:6800/43329', 'cluster_addr': '172.21.15.138:6801/43329', 'heartbeat_back_addr': '172.21.15.138:6805/43329', 'heartbeat_front_addr': '172.21.15.138:6803/43329', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '53ae8193-5c3e-4698-aacd-df2ad1db9143', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 15, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6812', 'nonce': 43351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6813', 'nonce': 43351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6815', 'nonce': 43351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.138:6814', 'nonce': 43351}]}, 'public_addr': '172.21.15.138:6812/43351', 'cluster_addr': '172.21.15.138:6813/43351', 'heartbeat_back_addr': '172.21.15.138:6815/43351', 'heartbeat_front_addr': '172.21.15.138:6814/43351', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:44.296659+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:23.214623+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-20T02:52:56.806719+0000', 'dead_epoch': 122}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:58.973108+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 49625, 'last_purged_snaps_scrub': '2024-04-20T02:52:38.383284+0000', 'dead_epoch': 150}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:29.483129+0000', 'dead_epoch': 26}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:18.346058+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-20T02:52:29.823916+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [6, 0, 2]}], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 0, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 63, 'length': 1}, {'begin': 65, 'length': 1}, {'begin': 69, 'length': 1}, {'begin': 71, 'length': 1}, {'begin': 75, 'length': 1}, {'begin': 77, 'length': 2}, {'begin': 80, 'length': 1}, {'begin': 83, 'length': 3}, {'begin': 90, 'length': 1}, {'begin': 92, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-20T02:53:29.602 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:53:29.603 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:29.600+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:29.604 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:29.604 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:29.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:29.604 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 140.11324418800001, 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.608 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 131.06306142099999, 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.609 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.610 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.611 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.612 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 131.06299597200001, 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:29.613 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.614 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:53:29.615 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.616 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.617 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 131.062970474, 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.618 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.619 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.620 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.621 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2524 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1723287~549075] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 101.275460991, 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054106970999999997, 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.622 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2524 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:29.623 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286419+0000", 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286924+0000", 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050494600000000004 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286944+0000", 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9833000000000001e-05 2024-04-20T02:53:29.624 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286957+0000", 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2901000000000001e-05 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298074+0000", 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011117320999999999 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298151+0000", 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6909999999999994e-05 2024-04-20T02:53:29.625 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298274+0000", 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012279700000000001 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326267+0000", 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027993284 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331226+0000", 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049590559999999999 2024-04-20T02:53:29.626 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340442+0000", 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092155740000000007 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340488+0000", 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6208000000000002e-05 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340528+0000", 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0277000000000001e-05 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.627 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3637 4.1 e137/60 4:80c7a726:::smithi10548823-19:head v 137'124, mlcod=120'120)", 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 53.937473431999997, 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052640605, 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.628 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624408+0000", 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624537+0000", 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012880199999999999 2024-04-20T02:53:29.629 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624548+0000", 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0216e-05 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624556+0000", 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6179999999999995e-06 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624603+0000", 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6601999999999999e-05 2024-04-20T02:53:29.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624632+0000", 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9277000000000001e-05 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.676999+0000", 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052366639999999999 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.677049+0000", 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0822999999999998e-05 2024-04-20T02:53:29.631 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3880 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'231, mlcod=141'228)", 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.154978452000002, 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.061291667000000001, 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:29.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406903+0000", 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407359+0000", 2024-04-20T02:53:29.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00045623700000000002 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407370+0000", 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1052e-05 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407377+0000", 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1080000000000004e-06 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407429+0000", 2024-04-20T02:53:29.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2033999999999997e-05 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407458+0000", 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8050000000000001e-05 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468161+0000", 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060703628000000003 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468196+0000", 2024-04-20T02:53:29.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4344999999999997e-05 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3888 4.7 4:e2c0d65b:::smithi10548823-46:head [write 413538~536345,stat] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.136425324000001, 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.059175945000000001, 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3888 2024-04-20T02:53:29.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425456+0000", 2024-04-20T02:53:29.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426097+0000", 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00064115099999999996 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426113+0000", 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6118000000000001e-05 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426122+0000", 2024-04-20T02:53:29.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.115e-06 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426179+0000", 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6815999999999999e-05 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426316+0000", 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000136606 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426499+0000", 2024-04-20T02:53:29.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000183191 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443652+0000", 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017153471999999999 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484036+0000", 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040384097000000001 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484231+0000", 2024-04-20T02:53:29.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000194811 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484363+0000", 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013146300000000001 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484426+0000", 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3563000000000001e-05 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484633+0000", 2024-04-20T02:53:29.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020662299999999999 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3889 4.7 4:e2c0d65b:::smithi10548823-46:head [write 1395221~583781] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.129531348999997, 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.057918795000000002, 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.642 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3889 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432350+0000", 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433490+0000", 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001140606 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433506+0000", 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5828000000000002e-05 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.644 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433515+0000", 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8279999999999992e-06 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443692+0000", 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010177222 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443839+0000", 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014662 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.444072+0000", 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023362200000000001 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484114+0000", 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040041641000000003 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484244+0000", 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012987200000000001 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484279+0000", 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5301e-05 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490055+0000", 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057763459999999999 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490115+0000", 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9320999999999997e-05 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490270+0000", 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015498199999999999 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3890 4.7 4:e2c0d65b:::smithi10548823-46:head [write 2556103~658617] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.121517248000004, 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066878325000000002, 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.648 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3890 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440363+0000", 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441831+0000", 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001467632 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441842+0000", 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0540000000000001e-05 2024-04-20T02:53:29.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441850+0000", 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7260000000000008e-06 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463536+0000", 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02168544 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463681+0000", 2024-04-20T02:53:29.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000145086 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463941+0000", 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00025968600000000002 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484317+0000", 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020376452 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490318+0000", 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060012490000000002 2024-04-20T02:53:29.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490398+0000", 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9598000000000006e-05 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506988+0000", 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016589706999999999 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507066+0000", 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8299000000000004e-05 2024-04-20T02:53:29.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507243+0000", 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000177503 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3891 4.7 4:e2c0d65b:::smithi10548823-46:head [setxattr _header (57) in=7b,truncate 3613538] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.120021739000002, 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088590031, 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.654 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3891 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:29.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441859+0000", 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441864+0000", 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8069999999999999e-06 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441869+0000", 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0989999999999998e-06 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441875+0000", 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5820000000000001e-06 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484675+0000", 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042799867999999998 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484817+0000", 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014236899999999999 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485048+0000", 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023057199999999999 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490359+0000", 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053116819999999999 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.658 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506963+0000", 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016603562999999998 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507256+0000", 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00029288699999999998 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530316+0000", 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023060343000000001 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530356+0000", 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9301999999999997e-05 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530451+0000", 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5037999999999994e-05 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3892 4.7 4:e2c0d65b:::smithi10548823-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:29.660 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.120002051, 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088912482000000001, 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3892 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441883+0000", 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9579999999999999e-06 2024-04-20T02:53:29.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441889+0000", 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7499999999999997e-06 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441895+0000", 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485431+0000", 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043536321000000003 2024-04-20T02:53:29.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485537+0000", 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010612599999999999 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530489+0000", 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044951996000000001 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530598+0000", 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108937 2024-04-20T02:53:29.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530793+0000", 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019441 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3898 4.8 4:1c68e542:::smithi10548823-8:head [write 447152~403196] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.095658968000002, 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.056917473000000003, 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.665 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3898 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466222+0000", 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466722+0000", 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00049941299999999998 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466734+0000", 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2860999999999999e-05 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466743+0000", 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7159999999999992e-06 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466809+0000", 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6165000000000001e-05 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466928+0000", 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011869199999999999 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.467158+0000", 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000230251 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498338+0000", 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.03117996 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.522958+0000", 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024619318000000001 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523002+0000", 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4675999999999997e-05 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523082+0000", 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9803999999999997e-05 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523141+0000", 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8767000000000002e-05 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3899 4.8 4:1c68e542:::smithi10548823-8:head [write 1512185~587450] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 45.087731200999997, 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060518556000000001, 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.671 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3899 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474149+0000", 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474686+0000", 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053696199999999999 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474703+0000", 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.615e-05 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474713+0000", 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0829000000000001e-05 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478533+0000", 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003819687 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478640+0000", 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106974 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478766+0000", 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125859 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523803+0000", 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045036616000000002 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526432+0000", 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026291489999999999 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534589+0000", 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081572209999999992 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534623+0000", 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534670+0000", 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6968999999999997e-05 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.676 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4016 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 665643~697786] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.300562436, 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060676251, 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4016 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.677 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261318+0000", 2024-04-20T02:53:29.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262081+0000", 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076285299999999997 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262095+0000", 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3986e-05 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262118+0000", 2024-04-20T02:53:29.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2407000000000001e-05 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262164+0000", 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6248999999999997e-05 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262238+0000", 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4202000000000005e-05 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262363+0000", 2024-04-20T02:53:29.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012437799999999999 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314944+0000", 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052581687000000002 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317407+0000", 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024628300000000001 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321887+0000", 2024-04-20T02:53:29.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0044803020000000002 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321941+0000", 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3769e-05 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321996+0000", 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4988000000000003e-05 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4017 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 2039745~753122] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 41.292932638000003, 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053146383999999998, 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4017 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268948+0000", 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269678+0000", 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072975799999999995 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269693+0000", 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.453e-05 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269704+0000", 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1623e-05 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277190+0000", 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074859849999999997 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277276+0000", 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5809999999999993e-05 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277405+0000", 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012914600000000001 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314952+0000", 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037546931999999998 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317441+0000", 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024889840000000001 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322031+0000", 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004589465 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322059+0000", 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322096+0000", 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591999999999998e-05 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4074 4.11 4:8b91c759:::smithi10548823-5:head [write 1591248~408851] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:29.688 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.419965249000001, 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053876146, 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4074 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141916+0000", 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142740+0000", 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000823616 2024-04-20T02:53:29.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142750+0000", 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6150000000000003e-06 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142756+0000", 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9419999999999998e-06 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2061999999999997e-05 2024-04-20T02:53:29.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142815+0000", 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5050000000000004e-06 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174069+0000", 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031253933999999997 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174193+0000", 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012386 2024-04-20T02:53:29.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174438+0000", 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00024561400000000001 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187472+0000", 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013033375999999999 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.188278+0000", 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00080595500000000002 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195551+0000", 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072733340000000002 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195631+0000", 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9882999999999994e-05 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195793+0000", 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000162223 2024-04-20T02:53:29.694 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4075 4.11 4:8b91c759:::smithi10548823-5:head [setxattr _header (56) in=7b,truncate 2000099] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.419114317000002, 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084064655000000002, 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.695 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4075 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142767+0000", 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142778+0000", 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0709e-05 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142783+0000", 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4539999999999998e-06 2024-04-20T02:53:29.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142788+0000", 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2850000000000002e-06 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142845+0000", 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6919000000000002e-05 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142856+0000", 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0936e-05 2024-04-20T02:53:29.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186635+0000", 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043778897999999997 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186756+0000", 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121087 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186967+0000", 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021064599999999999 2024-04-20T02:53:29.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.189477+0000", 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025104929999999999 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.190395+0000", 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00091717399999999998 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226595+0000", 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036200195999999997 2024-04-20T02:53:29.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226661+0000", 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6465999999999997e-05 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226833+0000", 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000171471 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.701 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4076 4.11 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 37.419073310999998, 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084363853000000003, 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4076 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.702 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142808+0000", 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142811+0000", 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9229999999999998e-06 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142818+0000", 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4420000000000003e-06 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.707 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142823+0000", 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5199999999999997e-06 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142875+0000", 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1759999999999999e-05 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142880+0000", 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6639999999999997e-06 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187624+0000", 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044743949999999998 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187726+0000", 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0001022 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226882+0000", 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039155862 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227003+0000", 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120715 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227173+0000", 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017032899999999999 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4183 3.2 e154/49 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head v 154'234, mlcod=154'232)", 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:29.710 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 35.908944378000001, 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.063121930000000007, 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652936+0000", 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657053+0000", 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041172520000000001 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657067+0000", 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3865e-05 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.712 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657076+0000", 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5390000000000007e-06 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657121+0000", 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5114000000000002e-05 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657138+0000", 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7600000000000001e-05 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:29.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716021+0000", 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.058882663000000002 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716060+0000", 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8826999999999999e-05 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:29.714 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:29.715 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:29.716 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:29.717+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:29.818 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:29.818+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:29.859 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:29.859 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:29.859 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:29.859 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:29.874 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:29.958 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:29.958+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:30.019 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:30.020+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:30.043 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:30.043 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:30.044 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:30.044 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:30.054 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:30.070 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:53:30.072 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:53:30.083 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-04-20T02:53:29.479944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":227613024,"num_objects":183,"num_object_clones":11,"num_object_copies":549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":4063,"num_read_kb":3824275,"num_write":2462,"num_write_kb":625983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":289530630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":663472,"kb_used_data":451468,"kb_used_omap":9,"kb_used_meta":211894,"kb_avail":565567568,"statfs":{"total":579820584960,"available":579141189632,"internally_reserved":0,"allocated":462303232,"data_stored":490875879,"data_compressed":11559750,"data_compressed_allocated":12521472,"data_compressed_original":27983872,"omap_allocated":9619,"internal_metadata":216980077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8879456,"num_objects":-8,"num_object_clones":-5,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":-6,"num_read":138,"num_read_kb":163707,"num_write":47,"num_write_kb":13053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000893"},"pg_stats":[{"pgid":"4.1","version":"184'139","reported_seq":689,"reported_epoch":189,"state":"peering","last_fresh":"2024-04-20T02:53:28.152924+0000","last_change":"2024-04-20T02:53:28.153106+0000","last_active":"2024-04-20T02:53:26.147742+0000","last_peered":"2024-04-20T02:53:27.151170+0000","last_clean":"2024-04-20T02:53:27.151170+0000","last_became_active":"2024-04-20T02:52:54.150080+0000","last_became_peered":"2024-04-20T02:53:27.150437+0000","last_unstale":"2024-04-20T02:53:28.152924+0000","last_undegraded":"2024-04-20T02:53:28.152924+0000","last_fullsized":"2024-04-20T02:53:28.152924+0000","mapping_epoch":154,"log_start":"184'139","ondisk_log_start":"184'139","created":22,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'131","last_scrub_stamp":"2024-04-20T02:52:54.612743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.612743+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":368,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.273286+0000","scrub_duration":0.0022862500000000001,"objects_trimmed":1,"snaptrim_duration":0.011020622000000001,"stat_sum":{"num_bytes":8084332,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":252,"num_read_kb":318196,"num_write":63,"num_write_kb":15810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":28106829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":132,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.145564+0000","last_change":"2024-04-20T02:53:18.152503+0000","last_active":"2024-04-20T02:53:26.145564+0000","last_peered":"2024-04-20T02:53:26.145564+0000","last_clean":"2024-04-20T02:53:26.145564+0000","last_became_active":"2024-04-20T02:53:18.152172+0000","last_became_peered":"2024-04-20T02:53:18.152172+0000","last_unstale":"2024-04-20T02:53:26.145564+0000","last_undegraded":"2024-04-20T02:53:26.145564+0000","last_fullsized":"2024-04-20T02:53:26.145564+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:56.983760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.983760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"186'205","reported_seq":771,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.145907+0000","last_change":"2024-04-20T02:53:25.193033+0000","last_active":"2024-04-20T02:53:26.145907+0000","last_peered":"2024-04-20T02:53:26.145907+0000","last_clean":"2024-04-20T02:53:26.145907+0000","last_became_active":"2024-04-20T02:53:16.152778+0000","last_became_peered":"2024-04-20T02:53:16.152778+0000","last_unstale":"2024-04-20T02:53:26.145907+0000","last_undegraded":"2024-04-20T02:53:26.145907+0000","last_fullsized":"2024-04-20T02:53:26.145907+0000","mapping_epoch":176,"log_start":"184'200","ondisk_log_start":"184'200","created":22,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":196,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001820667,"objects_trimmed":0,"snaptrim_duration":0.042836615000000001,"stat_sum":{"num_bytes":3999223,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":210,"num_read_kb":255240,"num_write":83,"num_write_kb":23977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":16021054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":212,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.286630+0000","last_change":"2024-04-20T02:53:19.107136+0000","last_active":"2024-04-20T02:53:26.286630+0000","last_peered":"2024-04-20T02:53:26.286630+0000","last_clean":"2024-04-20T02:53:26.286630+0000","last_became_active":"2024-04-20T02:52:51.153987+0000","last_became_peered":"2024-04-20T02:52:51.153987+0000","last_unstale":"2024-04-20T02:53:26.286630+0000","last_undegraded":"2024-04-20T02:53:26.286630+0000","last_fullsized":"2024-04-20T02:53:26.286630+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:24.234001+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":208,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-20T02:53:23.154453+0000","last_change":"2024-04-20T02:52:53.937048+0000","last_active":"2024-04-20T02:53:23.154453+0000","last_peered":"2024-04-20T02:53:23.154453+0000","last_clean":"2024-04-20T02:53:23.154453+0000","last_became_active":"2024-04-20T02:52:51.154174+0000","last_became_peered":"2024-04-20T02:52:51.154174+0000","last_unstale":"2024-04-20T02:53:23.154453+0000","last_undegraded":"2024-04-20T02:53:23.154453+0000","last_fullsized":"2024-04-20T02:53:23.154453+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:53.936957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:53.936957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.656218+0000","scrub_duration":0.0013534090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"184'308","reported_seq":838,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-04-20T02:53:24.156843+0000","last_change":"2024-04-20T02:53:23.159920+0000","last_active":"2024-04-20T02:53:24.156843+0000","last_peered":"2024-04-20T02:53:24.156843+0000","last_clean":"2024-04-20T02:53:24.156843+0000","last_became_active":"2024-04-20T02:53:21.153338+0000","last_became_peered":"2024-04-20T02:53:21.153338+0000","last_unstale":"2024-04-20T02:53:24.156843+0000","last_undegraded":"2024-04-20T02:53:24.156843+0000","last_fullsized":"2024-04-20T02:53:24.156843+0000","mapping_epoch":47,"log_start":"182'304","ondisk_log_start":"182'304","created":22,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'243","last_scrub_stamp":"2024-04-20T02:52:54.597931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.597931+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":549,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:59.861555+0000","scrub_duration":0.0028458709999999998,"objects_trimmed":2,"snaptrim_duration":0.0095485269999999994,"stat_sum":{"num_bytes":14219019,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":400,"num_read_kb":549925,"num_write":84,"num_write_kb":24112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":19243742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.1","version":"184'260","reported_seq":591,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-20T02:53:23.207508+0000","last_change":"2024-04-20T02:53:08.766997+0000","last_active":"2024-04-20T02:53:23.207508+0000","last_peered":"2024-04-20T02:53:23.207508+0000","last_clean":"2024-04-20T02:53:23.207508+0000","last_became_active":"2024-04-20T02:52:52.148432+0000","last_became_peered":"2024-04-20T02:52:52.148432+0000","last_unstale":"2024-04-20T02:53:23.207508+0000","last_undegraded":"2024-04-20T02:53:23.207508+0000","last_fullsized":"2024-04-20T02:53:23.207508+0000","mapping_epoch":152,"log_start":"180'257","ondisk_log_start":"180'257","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":282,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:29.132762+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27776822,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":233,"num_read_kb":8230,"num_write":314,"num_write_kb":78242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":83,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-20T02:53:23.153683+0000","last_change":"2024-04-20T02:52:32.918930+0000","last_active":"2024-04-20T02:53:23.153683+0000","last_peered":"2024-04-20T02:53:23.153683+0000","last_clean":"2024-04-20T02:53:23.153683+0000","last_became_active":"2024-04-20T02:52:22.261878+0000","last_became_peered":"2024-04-20T02:52:22.261878+0000","last_unstale":"2024-04-20T02:53:23.153683+0000","last_undegraded":"2024-04-20T02:53:23.153683+0000","last_fullsized":"2024-04-20T02:53:23.153683+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:32.918813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.918813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:36.236915+0000","scrub_duration":0.0014288059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"186'153","reported_seq":716,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.330174+0000","last_change":"2024-04-20T02:53:25.201610+0000","last_active":"2024-04-20T02:53:26.330174+0000","last_peered":"2024-04-20T02:53:26.330174+0000","last_clean":"2024-04-20T02:53:26.330174+0000","last_became_active":"2024-04-20T02:53:16.151498+0000","last_became_peered":"2024-04-20T02:53:16.151498+0000","last_unstale":"2024-04-20T02:53:26.330174+0000","last_undegraded":"2024-04-20T02:53:26.330174+0000","last_fullsized":"2024-04-20T02:53:26.330174+0000","mapping_epoch":176,"log_start":"185'149","ondisk_log_start":"185'149","created":22,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'57","last_scrub_stamp":"2024-04-20T02:51:21.386796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:21.386796+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":236,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018878359999999999,"objects_trimmed":0,"snaptrim_duration":0.041643649999999997,"stat_sum":{"num_bytes":9717018,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":174,"num_read_kb":188998,"num_write":46,"num_write_kb":11974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22843153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.2","version":"186'273","reported_seq":586,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.145282+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:53:26.145282+0000","last_peered":"2024-04-20T02:53:26.145282+0000","last_clean":"2024-04-20T02:53:26.145282+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:53:26.145282+0000","last_undegraded":"2024-04-20T02:53:26.145282+0000","last_fullsized":"2024-04-20T02:53:26.145282+0000","mapping_epoch":49,"log_start":"179'270","ondisk_log_start":"179'270","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":278,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24268031,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":242,"num_read_kb":13906,"num_write":326,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":351,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-20T02:53:23.154055+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:53:23.154055+0000","last_peered":"2024-04-20T02:53:23.154055+0000","last_clean":"2024-04-20T02:53:23.154055+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:53:23.154055+0000","last_undegraded":"2024-04-20T02:53:23.154055+0000","last_fullsized":"2024-04-20T02:53:23.154055+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":189,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-20T02:53:27.147525+0000","last_change":"2024-04-20T02:53:08.364116+0000","last_active":"2024-04-20T02:53:27.147525+0000","last_peered":"2024-04-20T02:53:27.147525+0000","last_clean":"2024-04-20T02:53:27.147525+0000","last_became_active":"2024-04-20T02:52:22.265487+0000","last_became_peered":"2024-04-20T02:52:22.265487+0000","last_unstale":"2024-04-20T02:53:27.147525+0000","last_undegraded":"2024-04-20T02:53:27.147525+0000","last_fullsized":"2024-04-20T02:53:27.147525+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.792324+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"186'105","reported_seq":356,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.145638+0000","last_change":"2024-04-20T02:53:25.163140+0000","last_active":"2024-04-20T02:53:26.145638+0000","last_peered":"2024-04-20T02:53:26.145638+0000","last_clean":"2024-04-20T02:53:26.145638+0000","last_became_active":"2024-04-20T02:53:18.152319+0000","last_became_peered":"2024-04-20T02:53:18.152319+0000","last_unstale":"2024-04-20T02:53:26.145638+0000","last_undegraded":"2024-04-20T02:53:26.145638+0000","last_fullsized":"2024-04-20T02:53:26.145638+0000","mapping_epoch":178,"log_start":"184'101","ondisk_log_start":"184'101","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'86","last_scrub_stamp":"2024-04-20T02:52:32.901998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.901998+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":131,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:43.974508+0000","scrub_duration":0.0024332220000000001,"objects_trimmed":0,"snaptrim_duration":0.99220328899999999,"stat_sum":{"num_bytes":5395179,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":35504,"num_write":16,"num_write_kb":3789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1543958,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.0","version":"186'240","reported_seq":517,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.286906+0000","last_change":"2024-04-20T02:52:42.201016+0000","last_active":"2024-04-20T02:53:26.286906+0000","last_peered":"2024-04-20T02:53:26.286906+0000","last_clean":"2024-04-20T02:53:26.286906+0000","last_became_active":"2024-04-20T02:52:25.272853+0000","last_became_peered":"2024-04-20T02:52:25.272853+0000","last_unstale":"2024-04-20T02:53:26.286906+0000","last_undegraded":"2024-04-20T02:53:26.286906+0000","last_fullsized":"2024-04-20T02:53:26.286906+0000","mapping_epoch":126,"log_start":"184'237","ondisk_log_start":"184'237","created":21,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'188","last_scrub_stamp":"2024-04-20T02:52:42.200865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:52:42.200865+0000","objects_scrubbed":31,"log_size":3,"log_dups_size":249,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.320740+0000","scrub_duration":0.0052191850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19588392,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":197,"num_read_kb":5871,"num_write":290,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":15696807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":90,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.145390+0000","last_change":"2024-04-20T02:52:56.212555+0000","last_active":"2024-04-20T02:53:26.145390+0000","last_peered":"2024-04-20T02:53:26.145390+0000","last_clean":"2024-04-20T02:53:26.145390+0000","last_became_active":"2024-04-20T02:52:22.272822+0000","last_became_peered":"2024-04-20T02:52:22.272822+0000","last_unstale":"2024-04-20T02:53:26.145390+0000","last_undegraded":"2024-04-20T02:53:26.145390+0000","last_fullsized":"2024-04-20T02:53:26.145390+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:56.212430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.212430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:20.820102+0000","scrub_duration":0.0013799870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"184'130","reported_seq":714,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-20T02:53:27.147678+0000","last_change":"2024-04-20T02:53:23.161348+0000","last_active":"2024-04-20T02:53:27.147678+0000","last_peered":"2024-04-20T02:53:27.147678+0000","last_clean":"2024-04-20T02:53:27.147678+0000","last_became_active":"2024-04-20T02:52:46.141983+0000","last_became_peered":"2024-04-20T02:52:46.141983+0000","last_unstale":"2024-04-20T02:53:27.147678+0000","last_undegraded":"2024-04-20T02:53:27.147678+0000","last_fullsized":"2024-04-20T02:53:27.147678+0000","mapping_epoch":147,"log_start":"179'124","ondisk_log_start":"179'124","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'119","last_scrub_stamp":"2024-04-20T02:52:56.564632+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.564632+0000","objects_scrubbed":4,"log_size":6,"log_dups_size":310,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:22.552034+0000","scrub_duration":0.0028929759999999998,"objects_trimmed":1,"snaptrim_duration":0.010818365,"stat_sum":{"num_bytes":2840449,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":43301,"num_write":4,"num_write_kb":1274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.3","version":"184'235","reported_seq":531,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-20T02:53:23.173407+0000","last_change":"2024-04-20T02:53:18.737801+0000","last_active":"2024-04-20T02:53:23.173407+0000","last_peered":"2024-04-20T02:53:23.173407+0000","last_clean":"2024-04-20T02:53:23.173407+0000","last_became_active":"2024-04-20T02:53:11.150822+0000","last_became_peered":"2024-04-20T02:53:11.150822+0000","last_unstale":"2024-04-20T02:53:23.173407+0000","last_undegraded":"2024-04-20T02:53:23.173407+0000","last_fullsized":"2024-04-20T02:53:23.173407+0000","mapping_epoch":171,"log_start":"175'232","ondisk_log_start":"175'232","created":21,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":238,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.609708+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":203,"num_read_kb":6612,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":18551655,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":168,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-20T02:53:27.147651+0000","last_change":"2024-04-20T02:51:52.777359+0000","last_active":"2024-04-20T02:53:27.147651+0000","last_peered":"2024-04-20T02:53:27.147651+0000","last_clean":"2024-04-20T02:53:27.147651+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:53:27.147651+0000","last_undegraded":"2024-04-20T02:53:27.147651+0000","last_fullsized":"2024-04-20T02:53:27.147651+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:51:52.777234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:51:52.777234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012962270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"185'48","reported_seq":502,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.287127+0000","last_change":"2024-04-20T02:53:25.199567+0000","last_active":"2024-04-20T02:53:26.287127+0000","last_peered":"2024-04-20T02:53:26.287127+0000","last_clean":"2024-04-20T02:53:26.287127+0000","last_became_active":"2024-04-20T02:53:17.156757+0000","last_became_peered":"2024-04-20T02:53:17.156757+0000","last_unstale":"2024-04-20T02:53:26.287127+0000","last_undegraded":"2024-04-20T02:53:26.287127+0000","last_fullsized":"2024-04-20T02:53:26.287127+0000","mapping_epoch":141,"log_start":"169'45","ondisk_log_start":"169'45","created":22,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:52:56.000178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.000178+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015078069999999999,"objects_trimmed":0,"snaptrim_duration":0.039989539999999997,"stat_sum":{"num_bytes":2991712,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":26723,"num_write":19,"num_write_kb":5030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5278912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.2","version":"0'0","reported_seq":260,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-20T02:53:27.147548+0000","last_change":"2024-04-20T02:53:25.404436+0000","last_active":"2024-04-20T02:53:27.147548+0000","last_peered":"2024-04-20T02:53:27.147548+0000","last_clean":"2024-04-20T02:53:27.147548+0000","last_became_active":"2024-04-20T02:53:25.402753+0000","last_became_peered":"2024-04-20T02:53:25.402753+0000","last_unstale":"2024-04-20T02:53:27.147548+0000","last_undegraded":"2024-04-20T02:53:27.147548+0000","last_fullsized":"2024-04-20T02:53:27.147548+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_clean_scrub_stamp":"2024-04-20T02:52:43.686760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.573670+0000","scrub_duration":0.0018670150000000001,"objects_trimmed":0,"snaptrim_duration":0.001125371,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":192,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.286670+0000","last_change":"2024-04-20T02:52:18.363613+0000","last_active":"2024-04-20T02:53:26.286670+0000","last_peered":"2024-04-20T02:53:26.286670+0000","last_clean":"2024-04-20T02:53:26.286670+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:53:26.286670+0000","last_undegraded":"2024-04-20T02:53:26.286670+0000","last_fullsized":"2024-04-20T02:53:26.286670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:18.363433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:18.363433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:38.745528+0000","scrub_duration":0.0016555929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"186'174","reported_seq":672,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-20T02:53:27.147443+0000","last_change":"2024-04-20T02:53:23.150954+0000","last_active":"2024-04-20T02:53:27.147443+0000","last_peered":"2024-04-20T02:53:27.147443+0000","last_clean":"2024-04-20T02:53:27.147443+0000","last_became_active":"2024-04-20T02:52:13.145577+0000","last_became_peered":"2024-04-20T02:52:13.145577+0000","last_unstale":"2024-04-20T02:53:27.147443+0000","last_undegraded":"2024-04-20T02:53:27.147443+0000","last_fullsized":"2024-04-20T02:53:27.147443+0000","mapping_epoch":62,"log_start":"186'170","ondisk_log_start":"186'170","created":22,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'74","last_scrub_stamp":"2024-04-20T02:51:56.784299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.784299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":328,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015920019999999999,"objects_trimmed":0,"snaptrim_duration":0.00066820099999999999,"stat_sum":{"num_bytes":10729969,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":90,"num_read_kb":103553,"num_write":46,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.9","version":"188'503","reported_seq":1162,"reported_epoch":188,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:53:27.327109+0000","last_change":"2024-04-20T02:53:27.185138+0000","last_active":"2024-04-20T02:53:27.327109+0000","last_peered":"2024-04-20T02:53:27.327109+0000","last_clean":"2024-04-20T02:53:27.327109+0000","last_became_active":"2024-04-20T02:53:11.151224+0000","last_became_peered":"2024-04-20T02:53:11.151224+0000","last_unstale":"2024-04-20T02:53:27.327109+0000","last_undegraded":"2024-04-20T02:53:27.327109+0000","last_fullsized":"2024-04-20T02:53:27.327109+0000","mapping_epoch":171,"log_start":"187'499","ondisk_log_start":"187'499","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'294","last_scrub_stamp":"2024-04-20T02:52:16.813424+0000","last_deep_scrub":"98'280","last_deep_scrub_stamp":"2024-04-20T02:51:56.947305+0000","last_clean_scrub_stamp":"2024-04-20T02:52:16.813424+0000","objects_scrubbed":7,"log_size":4,"log_dups_size":1120,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:41.845382+0000","scrub_duration":0.0034018329999999999,"objects_trimmed":0,"snaptrim_duration":0.89827231600000002,"stat_sum":{"num_bytes":27720751,"num_objects":12,"num_object_clones":5,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":223,"num_read_kb":242421,"num_write":112,"num_write_kb":32076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":34386920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"}]},{"pgid":"4.a","version":"184'120","reported_seq":587,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.145075+0000","last_change":"2024-04-20T02:53:23.158044+0000","last_active":"2024-04-20T02:53:26.145075+0000","last_peered":"2024-04-20T02:53:26.145075+0000","last_clean":"2024-04-20T02:53:26.145075+0000","last_became_active":"2024-04-20T02:53:18.153426+0000","last_became_peered":"2024-04-20T02:53:18.153426+0000","last_unstale":"2024-04-20T02:53:26.145075+0000","last_undegraded":"2024-04-20T02:53:26.145075+0000","last_fullsized":"2024-04-20T02:53:26.145075+0000","mapping_epoch":178,"log_start":"177'116","ondisk_log_start":"177'116","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'53","last_scrub_stamp":"2024-04-20T02:52:05.810627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:05.810627+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":116,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:28.333446+0000","scrub_duration":0.0022334680000000002,"objects_trimmed":0,"snaptrim_duration":0.00042807000000000001,"stat_sum":{"num_bytes":6343443,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":204,"num_read_kb":230945,"num_write":64,"num_write_kb":16950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.b","version":"184'350","reported_seq":895,"reported_epoch":184,"state":"active+clean","last_fresh":"2024-04-20T02:53:23.158331+0000","last_change":"2024-04-20T02:53:23.158331+0000","last_active":"2024-04-20T02:53:23.158331+0000","last_peered":"2024-04-20T02:53:23.158331+0000","last_clean":"2024-04-20T02:53:23.158331+0000","last_became_active":"2024-04-20T02:53:11.147630+0000","last_became_peered":"2024-04-20T02:53:11.147630+0000","last_unstale":"2024-04-20T02:53:23.158331+0000","last_undegraded":"2024-04-20T02:53:23.158331+0000","last_fullsized":"2024-04-20T02:53:23.158331+0000","mapping_epoch":171,"log_start":"179'344","ondisk_log_start":"179'344","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'157","last_scrub_stamp":"2024-04-20T02:52:07.013316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:07.013316+0000","objects_scrubbed":4,"log_size":6,"log_dups_size":344,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:24.431814+0000","scrub_duration":0.002193338,"objects_trimmed":1,"snaptrim_duration":0.0080676189999999998,"stat_sum":{"num_bytes":11923149,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":372,"num_read_kb":444659,"num_write":178,"num_write_kb":48621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":23150965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.c","version":"178'159","reported_seq":640,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.145434+0000","last_change":"2024-04-20T02:53:23.167866+0000","last_active":"2024-04-20T02:53:26.145434+0000","last_peered":"2024-04-20T02:53:26.145434+0000","last_clean":"2024-04-20T02:53:26.145434+0000","last_became_active":"2024-04-20T02:53:11.148476+0000","last_became_peered":"2024-04-20T02:53:11.148476+0000","last_unstale":"2024-04-20T02:53:26.145434+0000","last_undegraded":"2024-04-20T02:53:26.145434+0000","last_fullsized":"2024-04-20T02:53:26.145434+0000","mapping_epoch":171,"log_start":"173'155","ondisk_log_start":"173'155","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'95","last_scrub_stamp":"2024-04-20T02:52:09.803923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:09.803923+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":155,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:32.262327+0000","scrub_duration":0.0026115750000000001,"objects_trimmed":0,"snaptrim_duration":0.010578568,"stat_sum":{"num_bytes":4773320,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":178,"num_read_kb":193067,"num_write":90,"num_write_kb":24779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":14051707,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.d","version":"188'288","reported_seq":891,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-04-20T02:53:27.286145+0000","last_change":"2024-04-20T02:53:25.198992+0000","last_active":"2024-04-20T02:53:27.286145+0000","last_peered":"2024-04-20T02:53:27.286145+0000","last_clean":"2024-04-20T02:53:27.286145+0000","last_became_active":"2024-04-20T02:52:46.144032+0000","last_became_peered":"2024-04-20T02:52:46.144032+0000","last_unstale":"2024-04-20T02:53:27.286145+0000","last_undegraded":"2024-04-20T02:53:27.286145+0000","last_fullsized":"2024-04-20T02:53:27.286145+0000","mapping_epoch":147,"log_start":"186'285","ondisk_log_start":"186'285","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'169","last_scrub_stamp":"2024-04-20T02:52:08.841360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:08.841360+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":285,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002314502,"objects_trimmed":0,"snaptrim_duration":0.039094669999999998,"stat_sum":{"num_bytes":8353244,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":439,"num_read_kb":540894,"num_write":164,"num_write_kb":44217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":27793283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.e","version":"184'124","reported_seq":632,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.145741+0000","last_change":"2024-04-20T02:53:25.168677+0000","last_active":"2024-04-20T02:53:26.145741+0000","last_peered":"2024-04-20T02:53:26.145741+0000","last_clean":"2024-04-20T02:53:26.145741+0000","last_became_active":"2024-04-20T02:53:18.149105+0000","last_became_peered":"2024-04-20T02:53:18.149105+0000","last_unstale":"2024-04-20T02:53:26.145741+0000","last_undegraded":"2024-04-20T02:53:26.145741+0000","last_fullsized":"2024-04-20T02:53:26.145741+0000","mapping_epoch":178,"log_start":"173'118","ondisk_log_start":"173'118","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'77","last_scrub_stamp":"2024-04-20T02:51:46.088382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:46.088382+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":118,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002170693,"objects_trimmed":0,"snaptrim_duration":0.99770691499999997,"stat_sum":{"num_bytes":3882450,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":196,"num_read_kb":274578,"num_write":59,"num_write_kb":18271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":10957139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.f","version":"163'44","reported_seq":487,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.286971+0000","last_change":"2024-04-20T02:53:25.198652+0000","last_active":"2024-04-20T02:53:26.286971+0000","last_peered":"2024-04-20T02:53:26.286971+0000","last_clean":"2024-04-20T02:53:26.286971+0000","last_became_active":"2024-04-20T02:52:54.156070+0000","last_became_peered":"2024-04-20T02:52:54.156070+0000","last_unstale":"2024-04-20T02:53:26.286971+0000","last_undegraded":"2024-04-20T02:53:26.286971+0000","last_fullsized":"2024-04-20T02:53:26.286971+0000","mapping_epoch":154,"log_start":"153'40","ondisk_log_start":"153'40","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'36","last_scrub_stamp":"2024-04-20T02:52:46.138400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:46.138400+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":40,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:09.709596+0000","scrub_duration":0.003388247,"objects_trimmed":0,"snaptrim_duration":0.038600218999999998,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":116380,"num_write":27,"num_write_kb":7507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1714357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.10","version":"184'144","reported_seq":801,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-04-20T02:53:26.146236+0000","last_change":"2024-04-20T02:53:25.192706+0000","last_active":"2024-04-20T02:53:26.146236+0000","last_peered":"2024-04-20T02:53:26.146236+0000","last_clean":"2024-04-20T02:53:26.146236+0000","last_became_active":"2024-04-20T02:52:46.140881+0000","last_became_peered":"2024-04-20T02:52:46.140881+0000","last_unstale":"2024-04-20T02:53:26.146236+0000","last_undegraded":"2024-04-20T02:53:26.146236+0000","last_fullsized":"2024-04-20T02:53:26.146236+0000","mapping_epoch":147,"log_start":"180'139","ondisk_log_start":"180'139","created":60,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":5,"last_scrub":"151'132","last_scrub_stamp":"2024-04-20T02:52:55.175686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:55.175686+0000","objects_scrubbed":3,"log_size":5,"log_dups_size":137,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.746787+0000","scrub_duration":0.0022427829999999999,"objects_trimmed":0,"snaptrim_duration":0.042499190999999999,"stat_sum":{"num_bytes":5546409,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":168,"num_read_kb":225172,"num_write":30,"num_write_kb":9028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":17455143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3f","length":"1"},{"start":"41","length":"1"},{"start":"45","length":"1"},{"start":"47","length":"1"},{"start":"4b","length":"1"},{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]}],"pool_stats":[{"poolid":4,"num_pg":17,"stat_sum":{"num_bytes":128637415,"num_objects":52,"num_object_clones":11,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":3072,"num_read_kb":3789556,"num_write":1085,"num_write_kb":299828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":222547162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":310050816,"data_stored":342049472,"data_compressed":493873,"data_compressed_allocated":434176,"data_compressed_original":1245184,"omap_allocated":2595,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":51,"acting":51,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98385241,"num_objects":129,"num_object_clones":0,"num_object_copies":387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":875,"num_read_kb":34619,"num_write":1218,"num_write_kb":323739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":66983468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":225456128,"data_stored":240666281,"data_compressed":12916900,"data_compressed_allocated":13950976,"data_compressed_original":29847552,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509483,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69324,"kb_used_data":34104,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302516,"statfs":{"total":96636764160,"available":96565776384,"internally_reserved":0,"allocated":34922496,"data_stored":34809965,"data_compressed":69385,"data_compressed_allocated":102400,"data_compressed_original":708608,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.442,"15min":0.442},"min":{"1min":0.27000000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.57699999999999996,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.46700000000000003,"15min":0.45900000000000002},"min":{"1min":0.32400000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.48699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.53200000000000003,"15min":0.47799999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.9369999999999994,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.68500000000000005,"15min":0.64800000000000002},"min":{"1min":0.32400000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":9.8119999999999994,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.41799999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.307}]},{"osd":3,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.61399999999999999,"15min":0.56200000000000006},"min":{"1min":0.35499999999999998,"5min":0.307,"15min":0.307},"max":{"1min":9.6440000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.443},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.56399999999999995,"15min":0.55000000000000004},"min":{"1min":0.29999999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.8599999999999999,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.434}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.435,"15min":0.443},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65500000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.59399999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.39000000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.441,"15min":0.45000000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57699999999999996,"5min":2.431,"15min":2.431},"last":0.371}]},{"osd":6,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.51500000000000001,"15min":0.52200000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.64600000000000002,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.57799999999999996,"15min":0.59399999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73299999999999998,"5min":1.599,"15min":1.599},"last":0.35799999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757280,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103388,"kb_used_data":67464,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94268452,"statfs":{"total":96636764160,"available":96530894848,"internally_reserved":0,"allocated":69083136,"data_stored":72511395,"data_compressed":2035535,"data_compressed_allocated":2179072,"data_compressed_original":4706304,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56599999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.68700000000000006}]},{"osd":1,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42299999999999999,"15min":0.41699999999999998},"min":{"1min":0.34699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.64500000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.47899999999999998,"15min":0.46100000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.437,"15min":0.434},"min":{"1min":0.34499999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.48399999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.625}]},{"osd":6,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.27900000000000003,"5min":0.252,"15min":0.252},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.432},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.441,"15min":0.437},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.56399999999999995}]}]},{"osd":2,"up_from":0,"seq":592705486859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125780,"kb_used_data":90560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94246060,"statfs":{"total":96636764160,"available":96507965440,"internally_reserved":0,"allocated":92733440,"data_stored":101151189,"data_compressed":1830696,"data_compressed_allocated":1974272,"data_compressed_original":4153344,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.375},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.35399999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.442}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.753,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.503},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.63600000000000001,"5min":1.097,"15min":1.097},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.53500000000000003,"15min":0.54900000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.56399999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.63}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.59299999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.57099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112020,"kb_used_data":76808,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94259820,"statfs":{"total":96636764160,"available":96522055680,"internally_reserved":0,"allocated":78651392,"data_stored":85797524,"data_compressed":2004303,"data_compressed_allocated":2166784,"data_compressed_original":4640768,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.64000000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.44800000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":1.198,"15min":1.198},"last":2.0899999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.35399999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.85899999999999999,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.52900000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.01,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.627}]},{"osd":5,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47199999999999998,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69899999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.53400000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.48799999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.69799999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137636,"kb_used_data":102424,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94234204,"statfs":{"total":96636764160,"available":96495824896,"internally_reserved":0,"allocated":104882176,"data_stored":111592466,"data_compressed":2833859,"data_compressed_allocated":3072000,"data_compressed_original":7077888,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.78700000000000003,"15min":0.84699999999999998},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.46000000000000002,"15min":0.439},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59099999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.47399999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.503}]},{"osd":4,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.625,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.40899999999999997}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.57799999999999996,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.66100000000000003,"5min":1.038,"15min":1.038},"last":0.61899999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.47999999999999998,"15min":0.46899999999999997},"min":{"1min":0.378,"5min":0.247,"15min":0.247},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.59799999999999998,"15min":0.624},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68000000000000005,"5min":3.98,"15min":3.98},"last":0.51500000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509479,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115324,"kb_used_data":80108,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94256516,"statfs":{"total":96636764160,"available":96518672384,"internally_reserved":0,"allocated":82030592,"data_stored":85013340,"data_compressed":2785972,"data_compressed_allocated":3026944,"data_compressed_original":6696960,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36499999999999999,"15min":0.36399999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.377,"15min":0.38200000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.81799999999999995,"5min":1.427,"15min":1.427},"last":0.25700000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.36499999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.42799999999999999,"15min":0.41199999999999998},"min":{"1min":0.23899999999999999,"5min":0.215,"15min":0.215},"max":{"1min":3.2730000000000001,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":0.311}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90500000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.497}]},{"osd":5,"last update":"Sat Apr 20 02:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69599999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.627}]},{"osd":6,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.5349999999999999,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.55400000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.878,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.68799999999999994,"15min":0.70899999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.3380000000000001,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.50800000000000001}]}]},{"osd":4,"up_from":0,"seq":717259538437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49651712,"data_stored":52701812,"data_compressed":2726284,"data_compressed_allocated":2936832,"data_compressed_original":6119424,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31961088,"data_stored":33949988,"data_compressed":1733638,"data_compressed_allocated":1871872,"data_compressed_original":3956736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12894208,"data_stored":13909415,"data_compressed":734140,"data_compressed_allocated":790528,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32792576,"data_stored":35107919,"data_compressed":1992777,"data_compressed_allocated":2154496,"data_compressed_original":4575232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18509824,"data_stored":19818665,"data_compressed":1037033,"data_compressed_allocated":1118208,"data_compressed_original":2482176,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32792576,"data_stored":35107919,"data_compressed":1992777,"data_compressed_allocated":2154496,"data_compressed_original":4575232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46854144,"data_stored":50070563,"data_compressed":2700251,"data_compressed_allocated":2924544,"data_compressed_original":6303744,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30326784,"data_stored":30861008,"data_compressed":54370,"data_compressed_allocated":57344,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57044992,"data_stored":64216256,"data_compressed":97058,"data_compressed_allocated":102400,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2595,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":43810816,"data_stored":49745297,"data_compressed":11526,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56590336,"data_stored":67896718,"data_compressed":95804,"data_compressed_allocated":53248,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":34230272,"data_stored":36453414,"data_compressed":42758,"data_compressed_allocated":24576,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53882880,"data_stored":58008030,"data_compressed":128290,"data_compressed_allocated":114688,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":34164736,"data_stored":34868749,"data_compressed":64067,"data_compressed_allocated":69632,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:53:30.086 INFO:tasks.thrashosds.thrasher:Setting 4.b to [2, 0, 3, 1, 7, 6] 2024-04-20T02:53:30.086 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.b', '2', '0', '3', '1', '7', '6'] 2024-04-20T02:53:30.086 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 4.b 2 0 3 1 7 6 2024-04-20T02:53:30.120 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:30.121+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:30.166 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:53:30.207 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:30.207 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:30.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:30.207 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-04-20T02:53:30.207 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:30.218 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:30.259 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:30.259+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:30.322 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:30.323+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:30.361 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:30.361 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:30.361 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:30.361 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:30.371 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:30.423 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:30.423+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:30.523 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:30.524+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:30.539 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:30.540 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:30.540 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:30.540 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:30.540 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:30.550 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:30.624 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:30.624+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:30.718 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:30.718 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:30.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:30.718 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [] 2024-04-20T02:53:30.719 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:30.725 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:30.725+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:30.728 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:30.826 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:30.827+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:30.859 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:30.859 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:30.859 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:30.859 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:30.870 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:30.931 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:30.932+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:30.997 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:30.998 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:30.998 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:30.998 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:30.998 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:31.008 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:31.032 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:31.032+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:31.133 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:31.133+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:31.137 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 160.204277709, 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.138 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:53:31.140 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.141 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:53:31.142 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:31.143 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:31.144 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 160.20401750299999, 2024-04-20T02:53:31.144 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:53:31.144 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.144 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.144 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.144 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.144 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.144 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:53:31.144 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.144 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.145 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:53:31.146 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:53:31.147 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:53:31.148 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.149 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 160.20399603499999, 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.150 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:53:31.151 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.152 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.153 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3771 4.b 4:d4308fef:::smithi10548823-6:head [write 2548663~763022] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:31.154 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.954967377999999, 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072733780999999997, 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3771 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.155 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181891+0000", 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182301+0000", 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040985099999999998 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182317+0000", 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6305999999999999e-05 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182329+0000", 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1337e-05 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198720+0000", 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016391606 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198807+0000", 2024-04-20T02:53:31.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7319999999999997e-05 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198925+0000", 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117345 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218062+0000", 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019137134 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234990+0000", 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016928367 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254488+0000", 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019497460000000001 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254571+0000", 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3029999999999996e-05 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254627+0000", 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5865000000000001e-05 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3772 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3395047] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.954520805999998, 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072419708999999999, 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.161 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3772 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182343+0000", 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0099999999999997e-06 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182349+0000", 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9569999999999999e-06 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182354+0000", 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.826e-06 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217228+0000", 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034874555000000002 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217312+0000", 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3478000000000003e-05 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217431+0000", 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119512 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234969+0000", 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017537833999999999 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235012+0000", 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3411000000000002e-05 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254638+0000", 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019625815000000001 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254682+0000", 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3945e-05 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254759+0000", 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6882e-05 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.167 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3773 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.954502198999997, 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072538680999999994, 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3773 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:31.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182360+0000", 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3309999999999998e-06 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182366+0000", 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.981e-06 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182371+0000", 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5390000000000001e-06 2024-04-20T02:53:31.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217932+0000", 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035561700000000002 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218020+0000", 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7352e-05 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254778+0000", 2024-04-20T02:53:31.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036757971 2024-04-20T02:53:31.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.172 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254819+0000", 2024-04-20T02:53:31.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1525e-05 2024-04-20T02:53:31.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.172 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254897+0000", 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7621999999999996e-05 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4057 4.b 4:d4308fef:::smithi10548823-6:head [write 749678~790384,stat] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.993647459000002, 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074563737000000005, 2024-04-20T02:53:31.173 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.174 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:31.174 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.174 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.174 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.174 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4057 2024-04-20T02:53:31.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.174 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143210+0000", 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143760+0000", 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055015999999999995 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143772+0000", 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1345000000000001e-05 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143781+0000", 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0929999999999992e-06 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143895+0000", 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113998 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143906+0000", 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1195999999999999e-05 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.153590+0000", 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096836000000000005 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155611+0000", 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002021012 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217691+0000", 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062080244 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217777+0000", 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5583999999999999e-05 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.178 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4058 4.b 4:d4308fef:::smithi10548823-6:head [write 2332155~504118] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.993066433000003, 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074090323999999999, 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4058 2024-04-20T02:53:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143793+0000", 2024-04-20T02:53:31.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:31.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144104+0000", 2024-04-20T02:53:31.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000310971 2024-04-20T02:53:31.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144112+0000", 2024-04-20T02:53:31.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1990000000000006e-06 2024-04-20T02:53:31.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144118+0000", 2024-04-20T02:53:31.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.443e-06 2024-04-20T02:53:31.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144162+0000", 2024-04-20T02:53:31.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3066999999999999e-05 2024-04-20T02:53:31.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144172+0000", 2024-04-20T02:53:31.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0083e-05 2024-04-20T02:53:31.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.184 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.184 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155911+0000", 2024-04-20T02:53:31.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011739657000000001 2024-04-20T02:53:31.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:31.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155993+0000", 2024-04-20T02:53:31.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1754000000000004e-05 2024-04-20T02:53:31.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217810+0000", 2024-04-20T02:53:31.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061816958999999998 2024-04-20T02:53:31.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217884+0000", 2024-04-20T02:53:31.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4098999999999996e-05 2024-04-20T02:53:31.187 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.187 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.187 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.188 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4059 4.b 4:d4308fef:::smithi10548823-6:head [write 3271857~549469] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:31.188 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:31.188 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.992730272000003, 2024-04-20T02:53:31.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073860173000000001, 2024-04-20T02:53:31.188 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.188 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:31.189 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.189 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.189 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.189 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4059 2024-04-20T02:53:31.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.189 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:31.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:31.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144129+0000", 2024-04-20T02:53:31.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144439+0000", 2024-04-20T02:53:31.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030924000000000001 2024-04-20T02:53:31.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144447+0000", 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8930000000000005e-06 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144455+0000", 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1410000000000005e-06 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144505+0000", 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0698000000000001e-05 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144520+0000", 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4605e-05 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156034+0000", 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011513809 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156130+0000", 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6039999999999995e-05 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217915+0000", 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061785501999999999 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217990+0000", 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4770000000000004e-05 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.195 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4060 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3949678] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.992394892999997, 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085188106999999999, 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4060 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.196 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:31.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:31.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144469+0000", 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.946e-06 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144476+0000", 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7909999999999999e-06 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144482+0000", 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.601e-06 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144803+0000", 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032103699999999998 2024-04-20T02:53:31.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144813+0000", 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0183999999999999e-05 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156168+0000", 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011354523 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156254+0000", 2024-04-20T02:53:31.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5706999999999998e-05 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218023+0000", 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061769566999999997 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218137+0000", 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113912 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218377+0000", 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023941000000000001 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225107+0000", 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067304510000000001 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225121+0000", 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4341e-05 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229481+0000", 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043599880000000004 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229522+0000", 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0224e-05 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229653+0000", 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013190500000000001 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.203 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4061 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.992255276000002, 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085431572999999997, 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4061 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.204 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144604+0000", 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144608+0000", 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1080000000000003e-06 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144618+0000", 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5149999999999995e-06 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144625+0000", 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8260000000000004e-06 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144846+0000", 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022160699999999999 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144854+0000", 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0399999999999993e-06 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156327+0000", 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011472178 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156457+0000", 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013012599999999999 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218704+0000", 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062247080000000003 2024-04-20T02:53:31.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218795+0000", 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1391000000000006e-05 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229703+0000", 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010907834999999999 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229818+0000", 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115423 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.230037+0000", 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021803600000000001 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4146 3.1 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head [write 0~1129472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:31.210 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 38.956659299000002, 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067860845000000003, 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4146 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:31.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180199+0000", 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180439+0000", 2024-04-20T02:53:31.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000239375 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180449+0000", 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0220999999999999e-05 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180456+0000", 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0090000000000001e-06 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181170+0000", 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071358900000000004 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181331+0000", 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016178500000000001 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,5", 2024-04-20T02:53:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181490+0000", 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158703 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225260+0000", 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043770166999999999 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226398+0000", 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001138189 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247971+0000", 2024-04-20T02:53:31.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021572728999999999 2024-04-20T02:53:31.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248021+0000", 2024-04-20T02:53:31.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0176999999999997e-05 2024-04-20T02:53:31.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248062+0000", 2024-04-20T02:53:31.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0472000000000001e-05 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4623 4.3 4:c07c6daa:::smithi10548823-7:head [write 2772312~34523] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9867314510000007, 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074418926999999996, 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.217 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4623 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:31.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150128+0000", 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150143+0000", 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5166e-05 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150147+0000", 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.924e-06 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150151+0000", 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.777e-06 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150180+0000", 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8472000000000001e-05 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150184+0000", 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4349999999999999e-06 2024-04-20T02:53:31.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187063+0000", 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036878474000000001 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187145+0000", 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2107999999999994e-05 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187280+0000", 2024-04-20T02:53:31.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135384 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189223+0000", 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019430820000000001 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189411+0000", 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000188061 2024-04-20T02:53:31.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224429+0000", 2024-04-20T02:53:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035017386999999997 2024-04-20T02:53:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224487+0000", 2024-04-20T02:53:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8884000000000001e-05 2024-04-20T02:53:31.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224548+0000", 2024-04-20T02:53:31.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0238000000000002e-05 2024-04-20T02:53:31.225 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.225 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.225 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.226 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4624 4.3 4:c07c6daa:::smithi10548823-7:head [setxattr _header (56) in=7b,truncate 2806835] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:31.226 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:31.226 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9867052059999999, 2024-04-20T02:53:31.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074609524999999996, 2024-04-20T02:53:31.226 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.226 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.226 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.226 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.227 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.227 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4624 2024-04-20T02:53:31.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.227 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:31.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:31.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:31.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150158+0000", 2024-04-20T02:53:31.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9799999999999998e-06 2024-04-20T02:53:31.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150161+0000", 2024-04-20T02:53:31.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8140000000000001e-06 2024-04-20T02:53:31.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.230 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150164+0000", 2024-04-20T02:53:31.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5329999999999999e-06 2024-04-20T02:53:31.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150197+0000", 2024-04-20T02:53:31.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2926999999999998e-05 2024-04-20T02:53:31.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150201+0000", 2024-04-20T02:53:31.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2989999999999998e-06 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188464+0000", 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038262276999999997 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188576+0000", 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112413 2024-04-20T02:53:31.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188756+0000", 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000180206 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189459+0000", 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070307500000000005 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224557+0000", 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035097721999999998 2024-04-20T02:53:31.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224633+0000", 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5798999999999997e-05 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224690+0000", 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6784999999999999e-05 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224765+0000", 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4969999999999995e-05 2024-04-20T02:53:31.234 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4625 4.3 4:c07c6daa:::smithi10548823-7:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9866932439999996, 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074770387999999993, 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.235 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4625 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150169+0000", 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7680000000000001e-06 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150171+0000", 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8779999999999998e-06 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150175+0000", 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2909999999999999e-06 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150215+0000", 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0074000000000003e-05 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150219+0000", 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.177e-06 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189474+0000", 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039255274999999999 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189531+0000", 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7188999999999997e-05 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224802+0000", 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035270981999999999 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224861+0000", 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8276999999999998e-05 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224937+0000", 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6689999999999999e-05 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4661 4.3 4:cd374330:::smithi10548823-1:head [write 490870~618799] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9866824859999994, 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074896037999999998, 2024-04-20T02:53:31.240 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4661 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:31.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150452+0000", 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027429199999999998 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150458+0000", 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5810000000000002e-06 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150463+0000", 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7029999999999997e-06 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150486+0000", 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3308000000000001e-05 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150491+0000", 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6179999999999997e-06 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189565+0000", 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039073926000000002 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189713+0000", 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014765700000000001 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189930+0000", 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021749700000000001 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224591+0000", 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034661210999999997 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224775+0000", 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000184091 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224987+0000", 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021128100000000001 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225040+0000", 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3395e-05 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225074+0000", 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3699999999999999e-05 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.246 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4662 4.3 4:cd374330:::smithi10548823-1:head [write 1592676~774144] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9863916019999994, 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075836991000000006, 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4662 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.247 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150468+0000", 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150808+0000", 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033972899999999999 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150815+0000", 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8159999999999996e-06 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150819+0000", 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150855+0000", 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150886+0000", 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0876000000000001e-05 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.204837+0000", 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053950873000000003 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.204916+0000", 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9008e-05 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.205039+0000", 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012294000000000001 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224780+0000", 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019741771000000002 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225703+0000", 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00092290600000000003 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226196+0000", 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049284199999999995 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226250+0000", 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4413000000000002e-05 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226306+0000", 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.516e-05 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4663 4.3 4:cd374330:::smithi10548823-1:head [write 2793074~97796] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9860360289999992, 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075759200999999998, 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.253 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4663 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150883+0000", 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9308999999999997e-05 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150888+0000", 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1139999999999997e-06 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150892+0000", 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5899999999999999e-06 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150924+0000", 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2289000000000002e-05 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150929+0000", 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0259999999999998e-06 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221580+0000", 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070650828999999998 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221708+0000", 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012845700000000001 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221897+0000", 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000188789 2024-04-20T02:53:31.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225650+0000", 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037528980000000002 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226379+0000", 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072901200000000002 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226492+0000", 2024-04-20T02:53:31.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112905 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226526+0000", 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711000000000003e-05 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226583+0000", 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7624999999999998e-05 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4664 4.3 4:cd374330:::smithi10548823-1:head [setxattr _header (56) in=7b,truncate 2890870] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9859623989999999, 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076283431999999998, 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4664 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150900+0000", 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.571e-06 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150903+0000", 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9409999999999999e-06 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150907+0000", 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5599999999999998e-06 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150988+0000", 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1544000000000004e-05 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151006+0000", 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7754999999999999e-05 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225114+0000", 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074107889999999996 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225189+0000", 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4851999999999994e-05 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225305+0000", 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011592899999999999 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225800+0000", 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049498200000000004 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226624+0000", 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00082437000000000005 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227055+0000", 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043119499999999999 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227113+0000", 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7500999999999997e-05 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227181+0000", 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8625000000000001e-05 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4665 4.3 4:cd374330:::smithi10548823-1:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 9.9859380029999993, 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076413855000000003, 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:31.266 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4665 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150924+0000", 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8190000000000001e-06 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150929+0000", 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6400000000000002e-06 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150933+0000", 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5039999999999998e-06 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151025+0000", 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2354000000000004e-05 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151040+0000", 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708999999999999e-05 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225553+0000", 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074513356000000003 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225640+0000", 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6335000000000006e-05 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227206+0000", 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001566448 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227261+0000", 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4907000000000001e-05 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227336+0000", 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5050999999999997e-05 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:31.271 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:31.273 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:31.273 INFO:teuthology.orchestra.run.smithi105.stderr:set 4.b pg_upmap_items mapping to [2->0,3->1,7->6] 2024-04-20T02:53:31.274 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:53:31.275 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:31.273+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:31.373 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:31.373+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:31.441 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:31.441 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:31.442 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:31.442 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:31.452 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:31.493 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:31.493+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:31.576 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:31.576+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:31.604 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:53:31.624 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:31.624 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:31.624 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:31.624 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:31.624 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:31.636 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:31.677 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:31.677+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:31.753 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:31.773 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:31.776 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:31.777+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:31.792 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:31.809 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:31.809 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:31.811 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:31.820 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:31.832 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:31.852 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:31.873 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:31.878 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:31.878+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:31.892 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:31.894 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:31.894 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:31.894 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:31.894 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:31.894 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:31.894 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:31.894 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:31.894 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:31.962 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:31.963 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:31.972 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:31.979 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:31.979+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:32.080 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:32.080+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:32.101 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:32.101 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:32.112 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:32.181 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:32.181+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:32.242 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:32.252 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:32.296 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:32.297+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:32.382 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:32.382+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:32.396 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:32.396 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:32.407 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:32.482 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:32.483+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:32.575 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:32.575 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:32.584 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:32.584+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:32.585 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:32.685 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:32.686+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:32.754 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:32.755 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:32.765 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:32.806 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:32.806+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:32.887 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:32.888+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:32.934 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:32.934 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:32.944 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:32.988 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:32.988+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:33.072 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:33.072 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:33.082 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:33.089 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:33.089+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:33.190 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:33.191+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:33.211 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:33.211 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:33.222 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:33.275 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:53:33.320 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:33.320+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:33.391 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:33.391+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:33.393 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:33.394 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:33.404 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:33.493 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:33.493+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:33.533 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:33.534 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:33.544 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:33.628 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:33.629+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:33.695 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:33.696+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:33.711 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:33.712 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:33.722 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:33.796 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:33.796+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:33.850 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:33.851 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:33.860 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:33.902 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:33.902+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:33.906 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:33.997 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:33.998+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:34.031 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:34.032 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:34.043 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:34.092 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:34.099 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:34.099+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:34.113 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:34.151 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:34.170 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:34.190 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:34.201 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:34.201+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:34.210 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:34.229 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:34.229 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:34.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:34.230 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-04-20T02:53:34.230 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:34.230 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:34.240 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:34.249 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:34.250 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:34.250 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:34.250 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:34.250 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:34.250 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:34.251 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:34.251 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:34.251 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:34.301 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:34.302+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:34.307 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:53:34.375 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:34.375 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:34.375 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:34.375 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:34.385 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:34.427 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:34.427+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:34.503 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:34.503+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:34.554 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:34.554 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:34.554 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:34.555 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:34.555 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:34.565 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:34.604 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:34.605+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:34.692 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:34.693 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:34.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:34.693 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [] 2024-04-20T02:53:34.693 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:34.702 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:34.704 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:34.705+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:34.807 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:34.807+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:34.834 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:34.834 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:34.834 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:34.834 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:34.844 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:34.908 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:34.908+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:35.011 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:35.011+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:35.012 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:35.012 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:35.013 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:35.013 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:35.013 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:35.023 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:35.111 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:35.112+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 164.22112185200001, 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.155 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.156 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.157 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:53:35.158 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.159 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.160 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 164.220861646, 2024-04-20T02:53:35.161 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.162 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.163 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.164 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.165 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.166 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 164.220840178, 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.167 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.168 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:53:35.169 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:53:35.170 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:53:35.171 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3771 4.b 4:d4308fef:::smithi10548823-6:head [write 2548663~763022] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.971811520999999, 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072733780999999997, 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.172 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3771 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.173 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181891+0000", 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182301+0000", 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040985099999999998 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182317+0000", 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6305999999999999e-05 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.174 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182329+0000", 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1337e-05 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198720+0000", 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016391606 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198807+0000", 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7319999999999997e-05 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.175 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198925+0000", 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117345 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218062+0000", 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019137134 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234990+0000", 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016928367 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.176 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254488+0000", 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019497460000000001 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254571+0000", 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3029999999999996e-05 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254627+0000", 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5865000000000001e-05 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.177 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3772 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3395047] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.971364948999998, 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072419708999999999, 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3772 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.178 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.179 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182343+0000", 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0099999999999997e-06 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182349+0000", 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9569999999999999e-06 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182354+0000", 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.826e-06 2024-04-20T02:53:35.180 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217228+0000", 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034874555000000002 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217312+0000", 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3478000000000003e-05 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217431+0000", 2024-04-20T02:53:35.181 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119512 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234969+0000", 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017537833999999999 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235012+0000", 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3411000000000002e-05 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.182 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254638+0000", 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019625815000000001 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254682+0000", 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3945e-05 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254759+0000", 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6882e-05 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.183 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3773 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.971346341999997, 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072538680999999994, 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3773 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.185 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182360+0000", 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3309999999999998e-06 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182366+0000", 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.981e-06 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182371+0000", 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5390000000000001e-06 2024-04-20T02:53:35.186 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217932+0000", 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035561700000000002 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218020+0000", 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7352e-05 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254778+0000", 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036757971 2024-04-20T02:53:35.187 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254819+0000", 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1525e-05 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254897+0000", 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7621999999999996e-05 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4057 4.b 4:d4308fef:::smithi10548823-6:head [write 749678~790384,stat] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.188 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.010491602000002, 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074563737000000005, 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4057 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:35.189 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143210+0000", 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.190 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143760+0000", 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055015999999999995 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143772+0000", 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1345000000000001e-05 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143781+0000", 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0929999999999992e-06 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143895+0000", 2024-04-20T02:53:35.191 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113998 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143906+0000", 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1195999999999999e-05 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.153590+0000", 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096836000000000005 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155611+0000", 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002021012 2024-04-20T02:53:35.192 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217691+0000", 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062080244 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217777+0000", 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5583999999999999e-05 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4058 4.b 4:d4308fef:::smithi10548823-6:head [write 2332155~504118] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.193 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.009910576000003, 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074090323999999999, 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4058 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:35.194 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143793+0000", 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.195 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144104+0000", 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000310971 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144112+0000", 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1990000000000006e-06 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144118+0000", 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.443e-06 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.196 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144162+0000", 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3066999999999999e-05 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144172+0000", 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0083e-05 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155911+0000", 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011739657000000001 2024-04-20T02:53:35.197 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155993+0000", 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1754000000000004e-05 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217810+0000", 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061816958999999998 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217884+0000", 2024-04-20T02:53:35.198 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4098999999999996e-05 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4059 4.b 4:d4308fef:::smithi10548823-6:head [write 3271857~549469] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.009574415000003, 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073860173000000001, 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.199 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4059 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:35.200 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144129+0000", 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144439+0000", 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030924000000000001 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144447+0000", 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8930000000000005e-06 2024-04-20T02:53:35.201 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144455+0000", 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1410000000000005e-06 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144505+0000", 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0698000000000001e-05 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144520+0000", 2024-04-20T02:53:35.202 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4605e-05 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156034+0000", 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011513809 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156130+0000", 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6039999999999995e-05 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217915+0000", 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061785501999999999 2024-04-20T02:53:35.203 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217990+0000", 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4770000000000004e-05 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4060 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3949678] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.009239035999997, 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085188106999999999, 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.204 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4060 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:35.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144469+0000", 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.946e-06 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144476+0000", 2024-04-20T02:53:35.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7909999999999999e-06 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144482+0000", 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.601e-06 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144803+0000", 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032103699999999998 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144813+0000", 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0183999999999999e-05 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156168+0000", 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011354523 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156254+0000", 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5706999999999998e-05 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218023+0000", 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061769566999999997 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218137+0000", 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113912 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218377+0000", 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023941000000000001 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225107+0000", 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067304510000000001 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225121+0000", 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4341e-05 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229481+0000", 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043599880000000004 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229522+0000", 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0224e-05 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229653+0000", 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013190500000000001 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4061 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.211 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.009099419000002, 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085431572999999997, 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4061 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144604+0000", 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144608+0000", 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1080000000000003e-06 2024-04-20T02:53:35.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144618+0000", 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5149999999999995e-06 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144625+0000", 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8260000000000004e-06 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144846+0000", 2024-04-20T02:53:35.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022160699999999999 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144854+0000", 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0399999999999993e-06 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156327+0000", 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011472178 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156457+0000", 2024-04-20T02:53:35.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013012599999999999 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218704+0000", 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062247080000000003 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218795+0000", 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1391000000000006e-05 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229703+0000", 2024-04-20T02:53:35.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010907834999999999 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229818+0000", 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115423 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.230037+0000", 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021803600000000001 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4146 3.1 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head [write 0~1129472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.973503442000002, 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067860845000000003, 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4146 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180199+0000", 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180439+0000", 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000239375 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180449+0000", 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0220999999999999e-05 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180456+0000", 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0090000000000001e-06 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181170+0000", 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071358900000000004 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181331+0000", 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016178500000000001 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,5", 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181490+0000", 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158703 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225260+0000", 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043770166999999999 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226398+0000", 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001138189 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247971+0000", 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021572728999999999 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248021+0000", 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0176999999999997e-05 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248062+0000", 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0472000000000001e-05 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4623 4.3 4:c07c6daa:::smithi10548823-7:head [write 2772312~34523] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:35.223 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.003575594000001, 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074418926999999996, 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4623 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150128+0000", 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150143+0000", 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5166e-05 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150147+0000", 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.924e-06 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150151+0000", 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.777e-06 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150180+0000", 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8472000000000001e-05 2024-04-20T02:53:35.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150184+0000", 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4349999999999999e-06 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187063+0000", 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036878474000000001 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187145+0000", 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2107999999999994e-05 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187280+0000", 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135384 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189223+0000", 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019430820000000001 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189411+0000", 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000188061 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224429+0000", 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035017386999999997 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224487+0000", 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8884000000000001e-05 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224548+0000", 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0238000000000002e-05 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4624 4.3 4:c07c6daa:::smithi10548823-7:head [setxattr _header (56) in=7b,truncate 2806835] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.003549349, 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074609524999999996, 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.230 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4624 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150158+0000", 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9799999999999998e-06 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150161+0000", 2024-04-20T02:53:35.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8140000000000001e-06 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150164+0000", 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5329999999999999e-06 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150197+0000", 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2926999999999998e-05 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150201+0000", 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2989999999999998e-06 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188464+0000", 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038262276999999997 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188576+0000", 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112413 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188756+0000", 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000180206 2024-04-20T02:53:35.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189459+0000", 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070307500000000005 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224557+0000", 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035097721999999998 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224633+0000", 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5798999999999997e-05 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224690+0000", 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6784999999999999e-05 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224765+0000", 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4969999999999995e-05 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.236 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4625 4.3 4:c07c6daa:::smithi10548823-7:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.003537387, 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074770387999999993, 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4625 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150169+0000", 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7680000000000001e-06 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150171+0000", 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8779999999999998e-06 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150175+0000", 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2909999999999999e-06 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150215+0000", 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0074000000000003e-05 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150219+0000", 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.177e-06 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189474+0000", 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039255274999999999 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:35.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189531+0000", 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7188999999999997e-05 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224802+0000", 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035270981999999999 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224861+0000", 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8276999999999998e-05 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.241 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224937+0000", 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6689999999999999e-05 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4661 4.3 4:cd374330:::smithi10548823-1:head [write 490870~618799] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.003526629, 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074896037999999998, 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.242 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4661 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150452+0000", 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027429199999999998 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150458+0000", 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5810000000000002e-06 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150463+0000", 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7029999999999997e-06 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150486+0000", 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3308000000000001e-05 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150491+0000", 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6179999999999997e-06 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189565+0000", 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039073926000000002 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189713+0000", 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014765700000000001 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189930+0000", 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021749700000000001 2024-04-20T02:53:35.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224591+0000", 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034661210999999997 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224775+0000", 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000184091 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224987+0000", 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021128100000000001 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225040+0000", 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3395e-05 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225074+0000", 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3699999999999999e-05 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.248 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4662 4.3 4:cd374330:::smithi10548823-1:head [write 1592676~774144] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.003235745, 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075836991000000006, 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4662 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:35.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150468+0000", 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150808+0000", 2024-04-20T02:53:35.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033972899999999999 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150815+0000", 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8159999999999996e-06 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150819+0000", 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.251 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150855+0000", 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150886+0000", 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0876000000000001e-05 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.204837+0000", 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053950873000000003 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.204916+0000", 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9008e-05 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.205039+0000", 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012294000000000001 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224780+0000", 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019741771000000002 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225703+0000", 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00092290600000000003 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226196+0000", 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049284199999999995 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226250+0000", 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4413000000000002e-05 2024-04-20T02:53:35.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226306+0000", 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.516e-05 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4663 4.3 4:cd374330:::smithi10548823-1:head [write 2793074~97796] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.002880171999999, 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075759200999999998, 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.255 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4663 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:35.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150883+0000", 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9308999999999997e-05 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150888+0000", 2024-04-20T02:53:35.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1139999999999997e-06 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150892+0000", 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5899999999999999e-06 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150924+0000", 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2289000000000002e-05 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150929+0000", 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0259999999999998e-06 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221580+0000", 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070650828999999998 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221708+0000", 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012845700000000001 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221897+0000", 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000188789 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225650+0000", 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037528980000000002 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226379+0000", 2024-04-20T02:53:35.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072901200000000002 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226492+0000", 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112905 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226526+0000", 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711000000000003e-05 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226583+0000", 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7624999999999998e-05 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4664 4.3 4:cd374330:::smithi10548823-1:head [setxattr _header (56) in=7b,truncate 2890870] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.002806542, 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076283431999999998, 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.262 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4664 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:35.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150900+0000", 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.571e-06 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.264 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150903+0000", 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9409999999999999e-06 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150907+0000", 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5599999999999998e-06 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150988+0000", 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1544000000000004e-05 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151006+0000", 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7754999999999999e-05 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225114+0000", 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074107889999999996 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225189+0000", 2024-04-20T02:53:35.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4851999999999994e-05 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225305+0000", 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011592899999999999 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225800+0000", 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049498200000000004 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226624+0000", 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00082437000000000005 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227055+0000", 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043119499999999999 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227113+0000", 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7500999999999997e-05 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227181+0000", 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8625000000000001e-05 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4665 4.3 4:cd374330:::smithi10548823-1:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 14.002782145999999, 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076413855000000003, 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.269 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4665 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:35.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150924+0000", 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8190000000000001e-06 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150929+0000", 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6400000000000002e-06 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150933+0000", 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5039999999999998e-06 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151025+0000", 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2354000000000004e-05 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151040+0000", 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708999999999999e-05 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225553+0000", 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074513356000000003 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225640+0000", 2024-04-20T02:53:35.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6335000000000006e-05 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227206+0000", 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001566448 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227261+0000", 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4907000000000001e-05 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227336+0000", 2024-04-20T02:53:35.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5050999999999997e-05 2024-04-20T02:53:35.275 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.275 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.275 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.275 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.275 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.275 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:35.276 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:35.277 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:35.277+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:35.377 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:35.378+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:35.406 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:35.406 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:35.406 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:35.406 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:35.417 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:35.479 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:35.480+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:35.544 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:35.544 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:35.544 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:35.544 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:35.544 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:35.554 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:35.598 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:35.598+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:35.681 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:35.681+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:35.685 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:35.685 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:35.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:35.685 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:35.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.685 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:35.685 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 169.099123768, 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:53:35.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.691 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 169.078394457, 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:35.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:53:35.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.697 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 169.06914270600001, 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:35.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.700 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:53:35.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.703 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 161.27214348999999, 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:35.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:53:35.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:53:35.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 161.26613870200001, 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:53:35.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:53:35.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4029 4.13 4:cf11ccc0:::smithi10548823-4:head [write 699257~564637,stat] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.128596186000003, 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.86467550400000004, 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4029 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.712 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555666+0000", 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555809+0000", 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014306900000000001 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555832+0000", 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3680999999999999e-05 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555846+0000", 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3654e-05 2024-04-20T02:53:35.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555913+0000", 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6736999999999995e-05 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556058+0000", 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014538100000000001 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556392+0000", 2024-04-20T02:53:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033393099999999998 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578661+0000", 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022268553999999999 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578702+0000", 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1533999999999999e-05 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578747+0000", 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4858999999999999e-05 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578811+0000", 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3520000000000002e-05 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605743+0000", 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026932418999999999 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420312+0000", 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81456849600000003 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420343+0000", 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1911e-05 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4030 4.13 4:cf11ccc0:::smithi10548823-4:head [write 1873062~640438] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.121683064999999, 2024-04-20T02:53:35.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85780235299999996, 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4030 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562579+0000", 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562714+0000", 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013475 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562728+0000", 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3516000000000001e-05 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562736+0000", 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5690000000000005e-06 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578835+0000", 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016099156 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578957+0000", 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121094 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.579123+0000", 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016647100000000001 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605727+0000", 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026603643999999999 2024-04-20T02:53:35.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605749+0000", 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2070999999999999e-05 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605799+0000", 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0417000000000002e-05 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605841+0000", 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2344e-05 2024-04-20T02:53:35.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605907+0000", 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5135999999999994e-05 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420373+0000", 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81446675899999998 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420383+0000", 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0108999999999999e-05 2024-04-20T02:53:35.724 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4031 4.13 4:cf11ccc0:::smithi10548823-4:head [write 3305941~593316] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.114121490000002, 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85027709799999995, 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.725 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4031 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570142+0000", 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570277+0000", 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135606 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570288+0000", 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0719e-05 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570294+0000", 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2519999999999996e-06 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592786+0000", 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022491220999999999 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592911+0000", 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012546599999999999 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.593118+0000", 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000207011 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605878+0000", 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012760298999999999 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605934+0000", 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5813999999999998e-05 2024-04-20T02:53:35.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605963+0000", 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9275e-05 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606612+0000", 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00064813999999999996 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613782+0000", 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071705670000000001 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420411+0000", 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.80662907399999995 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420420+0000", 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5790000000000004e-06 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4032 4.13 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (56) in=7b,truncate 3899257] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.113965061999998, 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85015831100000006, 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4032 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570302+0000", 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1700000000000001e-06 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570305+0000", 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.976e-06 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570308+0000", 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8899999999999999e-06 2024-04-20T02:53:35.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605985+0000", 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035677121999999999 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606066+0000", 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0797000000000006e-05 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606211+0000", 2024-04-20T02:53:35.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014569999999999999 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607013+0000", 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00080160599999999995 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607046+0000", 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2938999999999998e-05 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607086+0000", 2024-04-20T02:53:35.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0203000000000001e-05 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607157+0000", 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0621e-05 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613833+0000", 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066762269999999999 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420448+0000", 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.806615478 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420457+0000", 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9779999999999994e-06 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4033 4.13 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:35.738 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.113953188000004, 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85021192700000003, 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4033 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570312+0000", 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419999999999999e-06 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570316+0000", 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3730000000000001e-06 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570318+0000", 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6680000000000001e-06 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606541+0000", 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036222797000000001 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606598+0000", 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6629999999999998e-05 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420515+0000", 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81391740099999998 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420523+0000", 2024-04-20T02:53:35.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6799999999999993e-06 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4034 4.13 4:cfd1e4d4:::smithi10548823-9:head [rollback 3f] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.113943550000002, 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85017147900000001, 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.743 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4034 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570320+0000", 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570322+0000", 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.37e-06 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570325+0000", 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7800000000000001e-06 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570328+0000", 2024-04-20T02:53:35.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5379999999999998e-06 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606632+0000", 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036304075999999998 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606716+0000", 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3689000000000005e-05 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420484+0000", 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81376812700000001 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420492+0000", 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1179999999999992e-06 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4062 4.9 4:9c87d36d:::smithi10548823-24:head [write 766485~465633] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.534089768999998, 2024-04-20T02:53:35.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075029242999999995, 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4062 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150173+0000", 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150435+0000", 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000262064 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150445+0000", 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8160000000000005e-06 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150450+0000", 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1880000000000001e-06 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150484+0000", 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150490+0000", 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6810000000000003e-06 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161686+0000", 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011196187 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161840+0000", 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015389000000000001 2024-04-20T02:53:35.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.162103+0000", 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026333299999999998 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224927+0000", 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062823458999999998 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224993+0000", 2024-04-20T02:53:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6667000000000004e-05 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225059+0000", 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6117000000000004e-05 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225119+0000", 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9342000000000002e-05 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225204+0000", 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4844000000000003e-05 2024-04-20T02:53:35.753 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4063 4.9 4:9c87d36d:::smithi10548823-24:head [write 1990637~647776] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.533807095, 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075097435000000004, 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.754 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4063 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150746+0000", 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028951399999999999 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150753+0000", 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150757+0000", 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2300000000000002e-06 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150801+0000", 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3421000000000003e-05 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150811+0000", 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0581000000000001e-05 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177065+0000", 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02625367 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177164+0000", 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8900999999999994e-05 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177319+0000", 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000155437 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225048+0000", 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047728857 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225217+0000", 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168395 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225375+0000", 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157837 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225437+0000", 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2824999999999996e-05 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225555+0000", 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117169 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4064 4.9 4:9c87d36d:::smithi10548823-24:head [write 3299280~667205] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:35.760 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.533500341999996, 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112222372, 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4064 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150763+0000", 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151076+0000", 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000312355 2024-04-20T02:53:35.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151083+0000", 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1010000000000003e-06 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151091+0000", 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0409999999999998e-06 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151127+0000", 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:35.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151133+0000", 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1099999999999999e-06 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200197+0000", 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049064000000000003 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200315+0000", 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117855 2024-04-20T02:53:35.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200545+0000", 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022990700000000001 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225595+0000", 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025050224999999999 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262670+0000", 2024-04-20T02:53:35.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075460999999997 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262767+0000", 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6479999999999998e-05 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262835+0000", 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999999e-05 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262986+0000", 2024-04-20T02:53:35.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015129799999999999 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4065 4.9 4:9c87d36d:::smithi10548823-24:head [setxattr _header (57) in=7b,truncate 3966485] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.533160461999998, 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112193353, 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.767 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4065 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:35.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151103+0000", 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151107+0000", 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151111+0000", 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2309999999999998e-06 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151116+0000", 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4850000000000003e-06 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151168+0000", 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2673000000000002e-05 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151179+0000", 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0112000000000001e-05 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225611+0000", 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074432549000000001 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225689+0000", 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7612000000000002e-05 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225858+0000", 2024-04-20T02:53:35.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016945400000000001 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262650+0000", 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036792126000000001 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262678+0000", 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8132000000000001e-05 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.772 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263077+0000", 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039868299999999999 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263141+0000", 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3448999999999995e-05 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263297+0000", 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015649699999999999 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.773 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4066 4.9 4:9c87d36d:::smithi10548823-24:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.533143606000003, 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112663448, 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4066 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.774 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151120+0000", 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151123+0000", 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4389999999999999e-06 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151128+0000", 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2449999999999998e-06 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151132+0000", 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9049999999999999e-06 2024-04-20T02:53:35.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151198+0000", 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5988000000000006e-05 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151205+0000", 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8770000000000004e-06 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226171+0000", 2024-04-20T02:53:35.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074966844000000005 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226261+0000", 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9975999999999994e-05 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263337+0000", 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075863000000001 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263443+0000", 2024-04-20T02:53:35.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000105678 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263784+0000", 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034103200000000001 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4095 4.9 4:964390c0:::smithi10548823-36:head [write 450520~558231] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:35.779 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.533129535999997, 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112875926, 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4095 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151134+0000", 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151379+0000", 2024-04-20T02:53:35.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024486000000000001 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151385+0000", 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1460000000000001e-06 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151389+0000", 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6610000000000002e-06 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151413+0000", 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3611000000000001e-05 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151419+0000", 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8780000000000003e-06 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226302+0000", 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074883540999999998 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.783 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226460+0000", 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015792000000000001 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226599+0000", 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013939599999999999 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262685+0000", 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036086115000000002 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262726+0000", 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0448999999999998e-05 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263833+0000", 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001106685 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263884+0000", 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1530999999999998e-05 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264011+0000", 2024-04-20T02:53:35.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012641900000000001 2024-04-20T02:53:35.786 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.786 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.786 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4096 4.9 4:964390c0:::smithi10548823-36:head [write 1483047~515125] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.532870000999999, 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11703358799999999, 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.787 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4096 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151644+0000", 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025029799999999998 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151651+0000", 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.148e-06 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151655+0000", 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2769999999999999e-06 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151689+0000", 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711999999999998e-05 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151698+0000", 2024-04-20T02:53:35.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0359999999999995e-06 2024-04-20T02:53:35.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245111+0000", 2024-04-20T02:53:35.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093413164000000007 2024-04-20T02:53:35.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245232+0000", 2024-04-20T02:53:35.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000120214 2024-04-20T02:53:35.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245424+0000", 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019272500000000001 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263033+0000", 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017608461999999998 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266386+0000", 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033530980000000001 2024-04-20T02:53:35.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268218+0000", 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001832101 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268283+0000", 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5334999999999997e-05 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268428+0000", 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014449999999999999 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4097 4.9 4:964390c0:::smithi10548823-36:head [setxattr _header (57) in=7b,truncate 2050520] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.532602048000001, 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.117031099, 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.795 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4097 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151665+0000", 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.614e-06 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151670+0000", 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151675+0000", 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0560000000000004e-06 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151738+0000", 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2447999999999996e-05 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151753+0000", 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5007e-05 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.798 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264044+0000", 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112291562 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264165+0000", 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121081 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:35.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264421+0000", 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025568599999999998 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266133+0000", 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001711668 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266755+0000", 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00062224100000000005 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268484+0000", 2024-04-20T02:53:35.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001729169 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268545+0000", 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0365000000000002e-05 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268693+0000", 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014860000000000001 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4098 4.9 4:964390c0:::smithi10548823-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 43.532584565999997, 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11735205, 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4098 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:35.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151679+0000", 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151681+0000", 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3140000000000002e-06 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151686+0000", 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0209999999999999e-06 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151691+0000", 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4700000000000004e-06 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151779+0000", 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8399999999999994e-05 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151788+0000", 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7129999999999994e-06 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265294+0000", 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11350631899999999 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265391+0000", 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6861000000000007e-05 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268730+0000", 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003339102 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268841+0000", 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110708 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:35.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:35.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:35.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.269032+0000", 2024-04-20T02:53:35.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019057899999999999 2024-04-20T02:53:35.807 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.807 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.807 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.807 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:35.807 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:35.807 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:35.809 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:35.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:35.809+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:35.909 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:35.910+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:35.979 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:35.979 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:35.979 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:35.979 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:35.989 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:36.030 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:36.030+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:36.111 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:36.112+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:36.159 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:36.159 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:36.159 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:36.159 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:36.159 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:36.170 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:36.213 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:36.213+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:36.262 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:53:36.275 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 3, 6, 7, 5, 2, 4] 2024-04-20T02:53:36.275 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:53:36.276 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-04-20T02:53:36.276 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-04-20T02:53:36.276 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:53:36.319 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:53:36.349 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:36.349 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 169.80290126, 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:53:36.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:36.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:53:36.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:53:36.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:53:36.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:53:36.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 161.974352372, 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.356 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.361 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 161.95810019000001, 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:36.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 103.998401171, 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.366 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:53:36.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:53:36.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:53:36.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:53:36.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3872 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 779924~785847,stat] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.995997932000002, 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070878165000000007, 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.372 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3872 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350641+0000", 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351179+0000", 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053797199999999995 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351196+0000", 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.66e-05 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351209+0000", 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2952000000000001e-05 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351306+0000", 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7272e-05 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351322+0000", 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6416e-05 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358164+0000", 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068416019999999996 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358336+0000", 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017156499999999999 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358594+0000", 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025794499999999998 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389148+0000", 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030554773 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409688+0000", 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020539497 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420893+0000", 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011205503 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420934+0000", 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1171000000000003e-05 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420998+0000", 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3386999999999995e-05 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421521+0000", 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052313200000000005 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.378 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3873 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 2078257~580112] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.995414646, 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075737411000000004, 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3873 2024-04-20T02:53:36.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351225+0000", 2024-04-20T02:53:36.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351614+0000", 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038825099999999999 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351625+0000", 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1173e-05 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351634+0000", 2024-04-20T02:53:36.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0489999999999993e-06 2024-04-20T02:53:36.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351696+0000", 2024-04-20T02:53:36.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1885999999999997e-05 2024-04-20T02:53:36.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351708+0000", 2024-04-20T02:53:36.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2702e-05 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389193+0000", 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037484574999999999 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389303+0000", 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110188 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389563+0000", 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026034499999999999 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420846+0000", 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031283014999999997 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421592+0000", 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074529299999999995 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421632+0000", 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0677999999999999e-05 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426683+0000", 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050507119999999997 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426752+0000", 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8484000000000004e-05 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426964+0000", 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021193500000000001 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3874 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 3301257~278667] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.994979645000001, 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075720365999999997, 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.390 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3874 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351660+0000", 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351852+0000", 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019116900000000001 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351861+0000", 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6050000000000004e-06 2024-04-20T02:53:36.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351869+0000", 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4780000000000002e-06 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351916+0000", 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7222999999999998e-05 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351927+0000", 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0791000000000001e-05 2024-04-20T02:53:36.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409721+0000", 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057793887000000002 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409840+0000", 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119837 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.410063+0000", 2024-04-20T02:53:36.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022257200000000001 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421678+0000", 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011615448 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421721+0000", 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2191000000000003e-05 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.423989+0000", 2024-04-20T02:53:36.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022682879999999998 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427051+0000", 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003061785 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427157+0000", 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106684 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427381+0000", 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022416299999999999 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3875 4.10 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (57) in=7b,truncate 3579924] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.994758484000002, 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075851731000000006, 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.397 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3875 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351886+0000", 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.561e-06 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351893+0000", 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6100000000000002e-06 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351899+0000", 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8660000000000002e-06 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352002+0000", 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103811 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352012+0000", 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4110000000000002e-06 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421749+0000", 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.069736690000000004 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421861+0000", 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112587 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422120+0000", 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025878099999999997 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424547+0000", 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024270379999999999 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424960+0000", 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041259899999999998 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426257+0000", 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001297231 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427489+0000", 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012320949999999999 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427562+0000", 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2825999999999997e-05 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427734+0000", 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017230699999999999 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.403 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3876 4.10 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.994736578000001, 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076230629999999994, 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3876 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351906+0000", 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6350000000000002e-06 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351912+0000", 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8209999999999998e-06 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351918+0000", 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3839999999999996e-06 2024-04-20T02:53:36.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352035+0000", 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117207 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352043+0000", 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0169999999999996e-06 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422511+0000", 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070467815000000003 2024-04-20T02:53:36.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:36.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422619+0000", 2024-04-20T02:53:36.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010819600000000001 2024-04-20T02:53:36.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427772+0000", 2024-04-20T02:53:36.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005153398 2024-04-20T02:53:36.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427900+0000", 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012740399999999999 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.428135+0000", 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235138 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.409 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4067 4.c 4:392c1e2f:::smithi10548823-44:head [write 783292~540882] snapc 47=[47,45,3f] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.204044369000002, 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072910900000000001, 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4067 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:36.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142595+0000", 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142941+0000", 2024-04-20T02:53:36.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034585799999999999 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142954+0000", 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3491e-05 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142965+0000", 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1139e-05 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143003+0000", 2024-04-20T02:53:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7499000000000001e-05 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143011+0000", 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5490000000000006e-06 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156033+0000", 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013021576999999999 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156242+0000", 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020885500000000001 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156474+0000", 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195900000000001 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177835+0000", 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021361594000000001 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178072+0000", 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000237304 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.214963+0000", 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036890158999999999 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215046+0000", 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3399000000000006e-05 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215507+0000", 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046127700000000001 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4068 4.c 4:392c1e2f:::smithi10548823-44:head [write 1932766~507787] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.203665426000001, 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073156952999999997, 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.416 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4068 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143300+0000", 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000325225 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143308+0000", 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3860000000000007e-06 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143315+0000", 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7000000000000002e-06 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143360+0000", 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5475000000000001e-05 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143373+0000", 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2204e-05 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178109+0000", 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034736547999999999 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178222+0000", 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112959 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178452+0000", 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000229686 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215864+0000", 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037412038000000002 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215878+0000", 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4029000000000001e-05 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215929+0000", 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0738000000000001e-05 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215989+0000", 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0350999999999999e-05 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216132+0000", 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014332099999999999 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.422 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4069 4.c 4:392c1e2f:::smithi10548823-44:head [write 3210350~572013] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.203316612999998, 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108473219, 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4069 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:36.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143664+0000", 2024-04-20T02:53:36.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034090500000000002 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143673+0000", 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1119999999999996e-06 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143679+0000", 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3589999999999996e-06 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143724+0000", 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4974e-05 2024-04-20T02:53:36.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143734+0000", 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0275000000000001e-05 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197473+0000", 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053738971000000003 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197616+0000", 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014275900000000001 2024-04-20T02:53:36.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197834+0000", 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021806000000000001 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251517+0000", 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053683445000000003 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251579+0000", 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1255999999999998e-05 2024-04-20T02:53:36.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251590+0000", 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1399e-05 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251657+0000", 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7039999999999995e-05 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251797+0000", 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000140235 2024-04-20T02:53:36.428 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4070 4.c 4:392c1e2f:::smithi10548823-44:head [setxattr _header (57) in=7b,truncate 3983292] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.202953825000002, 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108344988, 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4070 2024-04-20T02:53:36.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143686+0000", 2024-04-20T02:53:36.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143690+0000", 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7579999999999998e-06 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143697+0000", 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0119999999999999e-06 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143702+0000", 2024-04-20T02:53:36.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.78e-06 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143758+0000", 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640999999999998e-05 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143766+0000", 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9519999999999994e-06 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216172+0000", 2024-04-20T02:53:36.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072406561999999994 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216295+0000", 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122452 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216531+0000", 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023635999999999999 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251815+0000", 2024-04-20T02:53:36.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035283760999999997 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251856+0000", 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1270999999999998e-05 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251892+0000", 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251929+0000", 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7055999999999999e-05 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252032+0000", 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103183 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4071 4.c 4:392c1e2f:::smithi10548823-44:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.202918625999999, 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108670215, 2024-04-20T02:53:36.435 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4071 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143721+0000", 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143727+0000", 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2510000000000002e-06 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143739+0000", 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2649e-05 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143749+0000", 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3340000000000008e-06 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143788+0000", 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9773e-05 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143796+0000", 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8720000000000002e-06 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216846+0000", 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073049823 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216948+0000", 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101525 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252070+0000", 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035122911999999999 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252180+0000", 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010907499999999999 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252392+0000", 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021285400000000001 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4106 4.c 4:3d524298:::smithi10548823-18:head [write 484134~708729] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.202885520999999, 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108955254, 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4106 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144188+0000", 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000433771 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144198+0000", 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4970000000000007e-06 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144205+0000", 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3409999999999997e-06 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145714+0000", 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015088110000000001 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145727+0000", 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2989e-05 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217011+0000", 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071283814000000001 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217168+0000", 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015707300000000001 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217397+0000", 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022881900000000001 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251870+0000", 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034473539999999997 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252459+0000", 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000589151 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252511+0000", 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1282999999999997e-05 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252569+0000", 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8294999999999997e-05 2024-04-20T02:53:36.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252711+0000", 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000141435 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4107 4.c 4:3d524298:::smithi10548823-18:head [write 1883658~200476] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.202423455999998, 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.10880477099999999, 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.447 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4107 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:36.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144216+0000", 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144346+0000", 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000129732 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144355+0000", 2024-04-20T02:53:36.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4300000000000006e-06 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144361+0000", 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9150000000000001e-06 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145754+0000", 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0013929649999999999 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145763+0000", 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2990000000000002e-06 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243874+0000", 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.098111462999999996 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243990+0000", 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0001156 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.244226+0000", 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235839 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252760+0000", 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085345439999999998 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252817+0000", 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6842000000000002e-05 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252829+0000", 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1817999999999999e-05 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252884+0000", 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4610000000000001e-05 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253022+0000", 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013853800000000001 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.453 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4108 4.c 4:3d524298:::smithi10548823-18:head [setxattr _header (57) in=7b,truncate 2084134] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.202275032000003, 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.120324168, 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4108 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.454 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144365+0000", 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:36.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144369+0000", 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8500000000000004e-06 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144374+0000", 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0080000000000002e-06 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144379+0000", 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8239999999999999e-06 2024-04-20T02:53:36.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145790+0000", 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014115449999999999 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145800+0000", 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0429999999999996e-06 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253101+0000", 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.107301173 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253210+0000", 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109146 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253456+0000", 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024599700000000002 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254623+0000", 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001166718 2024-04-20T02:53:36.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.258220+0000", 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003597135 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264486+0000", 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062665819999999997 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264558+0000", 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1364000000000003e-05 2024-04-20T02:53:36.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264690+0000", 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013227099999999999 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4109 4.c 4:3d524298:::smithi10548823-18:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.202257510000003, 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12051582399999999, 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.460 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4109 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:36.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144385+0000", 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2869999999999999e-06 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144392+0000", 2024-04-20T02:53:36.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5749999999999997e-06 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144397+0000", 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3589999999999999e-06 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145829+0000", 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00143191 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145838+0000", 2024-04-20T02:53:36.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254052+0000", 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108214029 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254156+0000", 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010343799999999999 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264726+0000", 2024-04-20T02:53:36.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010570144 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264796+0000", 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9886000000000001e-05 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264899+0000", 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103336 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.465 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:4146 3.1 e153/152 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head v 153'228, mlcod=0'0)", 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 44.164861324, 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066047175, 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181778+0000", 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181997+0000", 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021896 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182011+0000", 2024-04-20T02:53:36.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3339999999999999e-05 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182021+0000", 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.046e-05 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182068+0000", 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6931999999999997e-05 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182104+0000", 2024-04-20T02:53:36.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6321e-05 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247786+0000", 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065681787000000005 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247827+0000", 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0491000000000002e-05 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4183 3.2 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head [write 0~2392064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 42.695227152000001, 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064859071000000004, 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4183 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651396+0000", 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999819 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652285+0000", 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088949500000000002 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652306+0000", 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0812e-05 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652333+0000", 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7086000000000001e-05 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652373+0000", 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9690000000000001e-05 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652499+0000", 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012642299999999999 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652608+0000", 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108661 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.473 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.702325+0000", 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049717048999999999 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.711665+0000", 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093403370000000006 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716188+0000", 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004522997 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716227+0000", 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9240999999999999e-05 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716273+0000", 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5228000000000001e-05 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:36.475 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:36.476 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:36.348+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:36.477 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:36.478 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:36.478 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:36.478 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:36.478 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:36.479 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:36.480+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:36.496 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:36.525 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:36.561 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:36.580 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:36.580+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:36.597 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:36.619 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:36.619 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:36.630 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:36.677 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:53:36.680 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:53:36.681 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:36.681+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:36.690 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-04-20T02:53:36.145435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":224121424,"num_objects":181,"num_object_clones":10,"num_object_copies":543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":4097,"num_read_kb":3855139,"num_write":2470,"num_write_kb":627372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":289530630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":17,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":659480,"kb_used_data":447472,"kb_used_omap":9,"kb_used_meta":211894,"kb_avail":565571560,"statfs":{"total":579820584960,"available":579145277440,"internally_reserved":0,"allocated":458211328,"data_stored":486714693,"data_compressed":11470125,"data_compressed_allocated":12410880,"data_compressed_original":27955200,"omap_allocated":9619,"internal_metadata":216980077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4971737,"num_objects":-4,"num_object_clones":-2,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.665015"},"pg_stats":[{"pgid":"4.1","version":"184'139","reported_seq":703,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.156534+0000","last_change":"2024-04-20T02:53:29.153095+0000","last_active":"2024-04-20T02:53:32.156534+0000","last_peered":"2024-04-20T02:53:32.156534+0000","last_clean":"2024-04-20T02:53:32.156534+0000","last_became_active":"2024-04-20T02:53:29.152688+0000","last_became_peered":"2024-04-20T02:53:29.152688+0000","last_unstale":"2024-04-20T02:53:32.156534+0000","last_undegraded":"2024-04-20T02:53:32.156534+0000","last_fullsized":"2024-04-20T02:53:32.156534+0000","mapping_epoch":154,"log_start":"184'139","ondisk_log_start":"184'139","created":22,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'131","last_scrub_stamp":"2024-04-20T02:52:54.612743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.612743+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":368,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:59.591499+0000","scrub_duration":0.0022862500000000001,"objects_trimmed":1,"snaptrim_duration":0.011020622000000001,"stat_sum":{"num_bytes":8084332,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":252,"num_read_kb":318196,"num_write":63,"num_write_kb":15810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":28106829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":148,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.194997+0000","last_change":"2024-04-20T02:53:30.912852+0000","last_active":"2024-04-20T02:53:31.194997+0000","last_peered":"2024-04-20T02:53:31.194997+0000","last_clean":"2024-04-20T02:53:31.194997+0000","last_became_active":"2024-04-20T02:53:18.152172+0000","last_became_peered":"2024-04-20T02:53:18.152172+0000","last_unstale":"2024-04-20T02:53:31.194997+0000","last_undegraded":"2024-04-20T02:53:31.194997+0000","last_fullsized":"2024-04-20T02:53:31.194997+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:32.689737+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"186'205","reported_seq":786,"reported_epoch":193,"state":"peering","last_fresh":"2024-04-20T02:53:32.156419+0000","last_change":"2024-04-20T02:53:32.156622+0000","last_active":"2024-04-20T02:53:30.295880+0000","last_peered":"2024-04-20T02:53:31.155330+0000","last_clean":"2024-04-20T02:53:31.155330+0000","last_became_active":"2024-04-20T02:53:16.152778+0000","last_became_peered":"2024-04-20T02:53:31.151765+0000","last_unstale":"2024-04-20T02:53:32.156419+0000","last_undegraded":"2024-04-20T02:53:32.156419+0000","last_fullsized":"2024-04-20T02:53:32.156419+0000","mapping_epoch":176,"log_start":"186'205","ondisk_log_start":"186'205","created":22,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":343,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001820667,"objects_trimmed":0,"snaptrim_duration":0.042836615000000001,"stat_sum":{"num_bytes":9545632,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":378,"num_read_kb":480412,"num_write":113,"num_write_kb":33005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33476197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":217,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.148717+0000","last_change":"2024-04-20T02:53:19.107136+0000","last_active":"2024-04-20T02:53:31.148717+0000","last_peered":"2024-04-20T02:53:31.148717+0000","last_clean":"2024-04-20T02:53:31.148717+0000","last_became_active":"2024-04-20T02:52:51.153987+0000","last_became_peered":"2024-04-20T02:52:51.153987+0000","last_unstale":"2024-04-20T02:53:31.148717+0000","last_undegraded":"2024-04-20T02:53:31.148717+0000","last_fullsized":"2024-04-20T02:53:31.148717+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:24.234001+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":218,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-20T02:53:33.158608+0000","last_change":"2024-04-20T02:52:53.937048+0000","last_active":"2024-04-20T02:53:33.158608+0000","last_peered":"2024-04-20T02:53:33.158608+0000","last_clean":"2024-04-20T02:53:33.158608+0000","last_became_active":"2024-04-20T02:52:51.154174+0000","last_became_peered":"2024-04-20T02:52:51.154174+0000","last_unstale":"2024-04-20T02:53:33.158608+0000","last_undegraded":"2024-04-20T02:53:33.158608+0000","last_fullsized":"2024-04-20T02:53:33.158608+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:53.936957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:53.936957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.656218+0000","scrub_duration":0.0013534090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"187'315","reported_seq":862,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.849744+0000","last_change":"2024-04-20T02:53:23.159920+0000","last_active":"2024-04-20T02:53:32.849744+0000","last_peered":"2024-04-20T02:53:32.849744+0000","last_clean":"2024-04-20T02:53:32.849744+0000","last_became_active":"2024-04-20T02:53:21.153338+0000","last_became_peered":"2024-04-20T02:53:21.153338+0000","last_unstale":"2024-04-20T02:53:32.849744+0000","last_undegraded":"2024-04-20T02:53:32.849744+0000","last_fullsized":"2024-04-20T02:53:32.849744+0000","mapping_epoch":47,"log_start":"186'312","ondisk_log_start":"186'312","created":22,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'243","last_scrub_stamp":"2024-04-20T02:52:54.597931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.597931+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":557,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:59.861555+0000","scrub_duration":0.0028458709999999998,"objects_trimmed":2,"snaptrim_duration":0.0095485269999999994,"stat_sum":{"num_bytes":15467220,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":413,"num_read_kb":563626,"num_write":89,"num_write_kb":25274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":19243742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.1","version":"187'261","reported_seq":601,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.849689+0000","last_change":"2024-04-20T02:53:08.766997+0000","last_active":"2024-04-20T02:53:32.849689+0000","last_peered":"2024-04-20T02:53:32.849689+0000","last_clean":"2024-04-20T02:53:32.849689+0000","last_became_active":"2024-04-20T02:52:52.148432+0000","last_became_peered":"2024-04-20T02:52:52.148432+0000","last_unstale":"2024-04-20T02:53:32.849689+0000","last_undegraded":"2024-04-20T02:53:32.849689+0000","last_fullsized":"2024-04-20T02:53:32.849689+0000","mapping_epoch":152,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:29.132762+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":233,"num_read_kb":8230,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":93,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-20T02:53:33.158259+0000","last_change":"2024-04-20T02:52:32.918930+0000","last_active":"2024-04-20T02:53:33.158259+0000","last_peered":"2024-04-20T02:53:33.158259+0000","last_clean":"2024-04-20T02:53:33.158259+0000","last_became_active":"2024-04-20T02:52:22.261878+0000","last_became_peered":"2024-04-20T02:52:22.261878+0000","last_unstale":"2024-04-20T02:53:33.158259+0000","last_undegraded":"2024-04-20T02:53:33.158259+0000","last_fullsized":"2024-04-20T02:53:33.158259+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:32.918813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.918813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:36.236915+0000","scrub_duration":0.0014288059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"186'153","reported_seq":733,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.855215+0000","last_change":"2024-04-20T02:53:31.855215+0000","last_active":"2024-04-20T02:53:31.855215+0000","last_peered":"2024-04-20T02:53:31.855215+0000","last_clean":"2024-04-20T02:53:31.855215+0000","last_became_active":"2024-04-20T02:53:16.151498+0000","last_became_peered":"2024-04-20T02:53:16.151498+0000","last_unstale":"2024-04-20T02:53:31.855215+0000","last_undegraded":"2024-04-20T02:53:31.855215+0000","last_fullsized":"2024-04-20T02:53:31.855215+0000","mapping_epoch":176,"log_start":"185'149","ondisk_log_start":"185'149","created":22,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'153","last_scrub_stamp":"2024-04-20T02:53:31.855077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:31.855077+0000","objects_scrubbed":5,"log_size":4,"log_dups_size":236,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.406736+0000","scrub_duration":0.0052908479999999999,"objects_trimmed":0,"snaptrim_duration":0.041643649999999997,"stat_sum":{"num_bytes":9717018,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":174,"num_read_kb":188998,"num_write":46,"num_write_kb":11974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22843153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.2","version":"192'274","reported_seq":592,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.197190+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:53:31.197190+0000","last_peered":"2024-04-20T02:53:31.197190+0000","last_clean":"2024-04-20T02:53:31.197190+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:53:31.197190+0000","last_undegraded":"2024-04-20T02:53:31.197190+0000","last_fullsized":"2024-04-20T02:53:31.197190+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":243,"num_read_kb":13907,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":361,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-20T02:53:33.158612+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:53:33.158612+0000","last_peered":"2024-04-20T02:53:33.158612+0000","last_clean":"2024-04-20T02:53:33.158612+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:53:33.158612+0000","last_undegraded":"2024-04-20T02:53:33.158612+0000","last_fullsized":"2024-04-20T02:53:33.158612+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":194,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.156615+0000","last_change":"2024-04-20T02:53:08.364116+0000","last_active":"2024-04-20T02:53:32.156615+0000","last_peered":"2024-04-20T02:53:32.156615+0000","last_clean":"2024-04-20T02:53:32.156615+0000","last_became_active":"2024-04-20T02:52:22.265487+0000","last_became_peered":"2024-04-20T02:52:22.265487+0000","last_unstale":"2024-04-20T02:53:32.156615+0000","last_undegraded":"2024-04-20T02:53:32.156615+0000","last_fullsized":"2024-04-20T02:53:32.156615+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.792324+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"186'105","reported_seq":362,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.194830+0000","last_change":"2024-04-20T02:53:25.163140+0000","last_active":"2024-04-20T02:53:31.194830+0000","last_peered":"2024-04-20T02:53:31.194830+0000","last_clean":"2024-04-20T02:53:31.194830+0000","last_became_active":"2024-04-20T02:53:18.152319+0000","last_became_peered":"2024-04-20T02:53:18.152319+0000","last_unstale":"2024-04-20T02:53:31.194830+0000","last_undegraded":"2024-04-20T02:53:31.194830+0000","last_fullsized":"2024-04-20T02:53:31.194830+0000","mapping_epoch":178,"log_start":"184'101","ondisk_log_start":"184'101","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'86","last_scrub_stamp":"2024-04-20T02:52:32.901998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.901998+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":131,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:43.974508+0000","scrub_duration":0.0024332220000000001,"objects_trimmed":0,"snaptrim_duration":0.99220328899999999,"stat_sum":{"num_bytes":5395179,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":35504,"num_write":16,"num_write_kb":3789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1543958,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.0","version":"190'241","reported_seq":526,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.148688+0000","last_change":"2024-04-20T02:52:42.201016+0000","last_active":"2024-04-20T02:53:31.148688+0000","last_peered":"2024-04-20T02:53:31.148688+0000","last_clean":"2024-04-20T02:53:31.148688+0000","last_became_active":"2024-04-20T02:52:25.272853+0000","last_became_peered":"2024-04-20T02:52:25.272853+0000","last_unstale":"2024-04-20T02:53:31.148688+0000","last_undegraded":"2024-04-20T02:53:31.148688+0000","last_fullsized":"2024-04-20T02:53:31.148688+0000","mapping_epoch":126,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'188","last_scrub_stamp":"2024-04-20T02:52:42.200865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:52:42.200865+0000","objects_scrubbed":31,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.320740+0000","scrub_duration":0.0052191850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":201,"num_read_kb":5875,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":15696807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":95,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.145249+0000","last_change":"2024-04-20T02:52:56.212555+0000","last_active":"2024-04-20T02:53:31.145249+0000","last_peered":"2024-04-20T02:53:31.145249+0000","last_clean":"2024-04-20T02:53:31.145249+0000","last_became_active":"2024-04-20T02:52:22.272822+0000","last_became_peered":"2024-04-20T02:52:22.272822+0000","last_unstale":"2024-04-20T02:53:31.145249+0000","last_undegraded":"2024-04-20T02:53:31.145249+0000","last_fullsized":"2024-04-20T02:53:31.145249+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:56.212430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.212430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:20.820102+0000","scrub_duration":0.0013799870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"184'130","reported_seq":720,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.156629+0000","last_change":"2024-04-20T02:53:23.161348+0000","last_active":"2024-04-20T02:53:32.156629+0000","last_peered":"2024-04-20T02:53:32.156629+0000","last_clean":"2024-04-20T02:53:32.156629+0000","last_became_active":"2024-04-20T02:52:46.141983+0000","last_became_peered":"2024-04-20T02:52:46.141983+0000","last_unstale":"2024-04-20T02:53:32.156629+0000","last_undegraded":"2024-04-20T02:53:32.156629+0000","last_fullsized":"2024-04-20T02:53:32.156629+0000","mapping_epoch":147,"log_start":"179'124","ondisk_log_start":"179'124","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'119","last_scrub_stamp":"2024-04-20T02:52:56.564632+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.564632+0000","objects_scrubbed":4,"log_size":6,"log_dups_size":310,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:22.552034+0000","scrub_duration":0.0028929759999999998,"objects_trimmed":1,"snaptrim_duration":0.010818365,"stat_sum":{"num_bytes":2840449,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":43301,"num_write":4,"num_write_kb":1274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.3","version":"184'235","reported_seq":540,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.849639+0000","last_change":"2024-04-20T02:53:18.737801+0000","last_active":"2024-04-20T02:53:32.849639+0000","last_peered":"2024-04-20T02:53:32.849639+0000","last_clean":"2024-04-20T02:53:32.849639+0000","last_became_active":"2024-04-20T02:53:11.150822+0000","last_became_peered":"2024-04-20T02:53:11.150822+0000","last_unstale":"2024-04-20T02:53:32.849639+0000","last_undegraded":"2024-04-20T02:53:32.849639+0000","last_fullsized":"2024-04-20T02:53:32.849639+0000","mapping_epoch":171,"log_start":"175'232","ondisk_log_start":"175'232","created":21,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":238,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.609708+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":203,"num_read_kb":6612,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":18551655,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":184,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.389983+0000","last_change":"2024-04-20T02:53:32.389983+0000","last_active":"2024-04-20T02:53:32.389983+0000","last_peered":"2024-04-20T02:53:32.389983+0000","last_clean":"2024-04-20T02:53:32.389983+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:53:32.389983+0000","last_undegraded":"2024-04-20T02:53:32.389983+0000","last_fullsized":"2024-04-20T02:53:32.389983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:54.846797+0000","scrub_duration":0.0028872630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"185'48","reported_seq":508,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.148627+0000","last_change":"2024-04-20T02:53:25.199567+0000","last_active":"2024-04-20T02:53:31.148627+0000","last_peered":"2024-04-20T02:53:31.148627+0000","last_clean":"2024-04-20T02:53:31.148627+0000","last_became_active":"2024-04-20T02:53:17.156757+0000","last_became_peered":"2024-04-20T02:53:17.156757+0000","last_unstale":"2024-04-20T02:53:31.148627+0000","last_undegraded":"2024-04-20T02:53:31.148627+0000","last_fullsized":"2024-04-20T02:53:31.148627+0000","mapping_epoch":141,"log_start":"169'45","ondisk_log_start":"169'45","created":22,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:52:56.000178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.000178+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015078069999999999,"objects_trimmed":0,"snaptrim_duration":0.039989539999999997,"stat_sum":{"num_bytes":2991712,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":26723,"num_write":19,"num_write_kb":5030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5278912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.2","version":"0'0","reported_seq":266,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.156556+0000","last_change":"2024-04-20T02:53:25.404436+0000","last_active":"2024-04-20T02:53:32.156556+0000","last_peered":"2024-04-20T02:53:32.156556+0000","last_clean":"2024-04-20T02:53:32.156556+0000","last_became_active":"2024-04-20T02:53:25.402753+0000","last_became_peered":"2024-04-20T02:53:25.402753+0000","last_unstale":"2024-04-20T02:53:32.156556+0000","last_undegraded":"2024-04-20T02:53:32.156556+0000","last_fullsized":"2024-04-20T02:53:32.156556+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_clean_scrub_stamp":"2024-04-20T02:52:43.686760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.573670+0000","scrub_duration":0.0018670150000000001,"objects_trimmed":0,"snaptrim_duration":0.001125371,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":197,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.148458+0000","last_change":"2024-04-20T02:52:18.363613+0000","last_active":"2024-04-20T02:53:31.148458+0000","last_peered":"2024-04-20T02:53:31.148458+0000","last_clean":"2024-04-20T02:53:31.148458+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:53:31.148458+0000","last_undegraded":"2024-04-20T02:53:31.148458+0000","last_fullsized":"2024-04-20T02:53:31.148458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:18.363433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:18.363433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:38.745528+0000","scrub_duration":0.0016555929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"186'174","reported_seq":679,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-20T02:53:33.349960+0000","last_change":"2024-04-20T02:53:23.150954+0000","last_active":"2024-04-20T02:53:33.349960+0000","last_peered":"2024-04-20T02:53:33.349960+0000","last_clean":"2024-04-20T02:53:33.349960+0000","last_became_active":"2024-04-20T02:52:13.145577+0000","last_became_peered":"2024-04-20T02:52:13.145577+0000","last_unstale":"2024-04-20T02:53:33.349960+0000","last_undegraded":"2024-04-20T02:53:33.349960+0000","last_fullsized":"2024-04-20T02:53:33.349960+0000","mapping_epoch":62,"log_start":"186'170","ondisk_log_start":"186'170","created":22,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'74","last_scrub_stamp":"2024-04-20T02:51:56.784299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.784299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":328,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015920019999999999,"objects_trimmed":0,"snaptrim_duration":0.00066820099999999999,"stat_sum":{"num_bytes":10729969,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":90,"num_read_kb":103553,"num_write":46,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.9","version":"192'507","reported_seq":1176,"reported_epoch":192,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:53:31.194099+0000","last_change":"2024-04-20T02:53:31.192280+0000","last_active":"2024-04-20T02:53:31.194099+0000","last_peered":"2024-04-20T02:53:31.194099+0000","last_clean":"2024-04-20T02:53:31.194099+0000","last_became_active":"2024-04-20T02:53:11.151224+0000","last_became_peered":"2024-04-20T02:53:11.151224+0000","last_unstale":"2024-04-20T02:53:31.194099+0000","last_undegraded":"2024-04-20T02:53:31.194099+0000","last_fullsized":"2024-04-20T02:53:31.194099+0000","mapping_epoch":171,"log_start":"188'503","ondisk_log_start":"188'503","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'294","last_scrub_stamp":"2024-04-20T02:52:16.813424+0000","last_deep_scrub":"98'280","last_deep_scrub_stamp":"2024-04-20T02:51:56.947305+0000","last_clean_scrub_stamp":"2024-04-20T02:52:16.813424+0000","objects_scrubbed":7,"log_size":4,"log_dups_size":1124,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:41.845382+0000","scrub_duration":0.0034018329999999999,"objects_trimmed":0,"snaptrim_duration":0.041609871,"stat_sum":{"num_bytes":23290123,"num_objects":10,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":223,"num_read_kb":242421,"num_write":112,"num_write_kb":32076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":34386920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4d","length":"1"}]},{"pgid":"4.a","version":"184'120","reported_seq":593,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.194862+0000","last_change":"2024-04-20T02:53:23.158044+0000","last_active":"2024-04-20T02:53:31.194862+0000","last_peered":"2024-04-20T02:53:31.194862+0000","last_clean":"2024-04-20T02:53:31.194862+0000","last_became_active":"2024-04-20T02:53:18.153426+0000","last_became_peered":"2024-04-20T02:53:18.153426+0000","last_unstale":"2024-04-20T02:53:31.194862+0000","last_undegraded":"2024-04-20T02:53:31.194862+0000","last_fullsized":"2024-04-20T02:53:31.194862+0000","mapping_epoch":178,"log_start":"177'116","ondisk_log_start":"177'116","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'53","last_scrub_stamp":"2024-04-20T02:52:05.810627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:05.810627+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":116,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022334680000000002,"objects_trimmed":0,"snaptrim_duration":0.00042807000000000001,"stat_sum":{"num_bytes":6343443,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":204,"num_read_kb":230945,"num_write":64,"num_write_kb":16950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.b","version":"184'350","reported_seq":923,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.849671+0000","last_change":"2024-04-20T02:53:30.856293+0000","last_active":"2024-04-20T02:53:32.849671+0000","last_peered":"2024-04-20T02:53:32.849671+0000","last_clean":"2024-04-20T02:53:32.849671+0000","last_became_active":"2024-04-20T02:53:11.147630+0000","last_became_peered":"2024-04-20T02:53:11.147630+0000","last_unstale":"2024-04-20T02:53:32.849671+0000","last_undegraded":"2024-04-20T02:53:32.849671+0000","last_fullsized":"2024-04-20T02:53:32.849671+0000","mapping_epoch":171,"log_start":"179'344","ondisk_log_start":"179'344","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'350","last_scrub_stamp":"2024-04-20T02:53:30.856060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.856060+0000","objects_scrubbed":5,"log_size":6,"log_dups_size":344,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.470953+0000","scrub_duration":0.0031057490000000001,"objects_trimmed":1,"snaptrim_duration":0.0080676189999999998,"stat_sum":{"num_bytes":11923149,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":388,"num_read_kb":461817,"num_write":178,"num_write_kb":48621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":23150965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.c","version":"178'159","reported_seq":646,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.145477+0000","last_change":"2024-04-20T02:53:23.167866+0000","last_active":"2024-04-20T02:53:31.145477+0000","last_peered":"2024-04-20T02:53:31.145477+0000","last_clean":"2024-04-20T02:53:31.145477+0000","last_became_active":"2024-04-20T02:53:11.148476+0000","last_became_peered":"2024-04-20T02:53:11.148476+0000","last_unstale":"2024-04-20T02:53:31.145477+0000","last_undegraded":"2024-04-20T02:53:31.145477+0000","last_fullsized":"2024-04-20T02:53:31.145477+0000","mapping_epoch":171,"log_start":"173'155","ondisk_log_start":"173'155","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'95","last_scrub_stamp":"2024-04-20T02:52:09.803923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:09.803923+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":155,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:32.262327+0000","scrub_duration":0.0026115750000000001,"objects_trimmed":0,"snaptrim_duration":0.010578568,"stat_sum":{"num_bytes":4773320,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":178,"num_read_kb":193067,"num_write":90,"num_write_kb":24779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":14051707,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.d","version":"188'288","reported_seq":897,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.151054+0000","last_change":"2024-04-20T02:53:25.198992+0000","last_active":"2024-04-20T02:53:31.151054+0000","last_peered":"2024-04-20T02:53:31.151054+0000","last_clean":"2024-04-20T02:53:31.151054+0000","last_became_active":"2024-04-20T02:52:46.144032+0000","last_became_peered":"2024-04-20T02:52:46.144032+0000","last_unstale":"2024-04-20T02:53:31.151054+0000","last_undegraded":"2024-04-20T02:53:31.151054+0000","last_fullsized":"2024-04-20T02:53:31.151054+0000","mapping_epoch":147,"log_start":"186'285","ondisk_log_start":"186'285","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'169","last_scrub_stamp":"2024-04-20T02:52:08.841360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:08.841360+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":285,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002314502,"objects_trimmed":0,"snaptrim_duration":0.039094669999999998,"stat_sum":{"num_bytes":8353244,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":439,"num_read_kb":540894,"num_write":164,"num_write_kb":44217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":27793283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.e","version":"184'124","reported_seq":638,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.194854+0000","last_change":"2024-04-20T02:53:25.168677+0000","last_active":"2024-04-20T02:53:31.194854+0000","last_peered":"2024-04-20T02:53:31.194854+0000","last_clean":"2024-04-20T02:53:31.194854+0000","last_became_active":"2024-04-20T02:53:18.149105+0000","last_became_peered":"2024-04-20T02:53:18.149105+0000","last_unstale":"2024-04-20T02:53:31.194854+0000","last_undegraded":"2024-04-20T02:53:31.194854+0000","last_fullsized":"2024-04-20T02:53:31.194854+0000","mapping_epoch":178,"log_start":"173'118","ondisk_log_start":"173'118","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'77","last_scrub_stamp":"2024-04-20T02:51:46.088382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:46.088382+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":118,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002170693,"objects_trimmed":0,"snaptrim_duration":0.99770691499999997,"stat_sum":{"num_bytes":3882450,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":196,"num_read_kb":274578,"num_write":59,"num_write_kb":18271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":10957139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.f","version":"163'44","reported_seq":493,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.148891+0000","last_change":"2024-04-20T02:53:25.198652+0000","last_active":"2024-04-20T02:53:31.148891+0000","last_peered":"2024-04-20T02:53:31.148891+0000","last_clean":"2024-04-20T02:53:31.148891+0000","last_became_active":"2024-04-20T02:52:54.156070+0000","last_became_peered":"2024-04-20T02:52:54.156070+0000","last_unstale":"2024-04-20T02:53:31.148891+0000","last_undegraded":"2024-04-20T02:53:31.148891+0000","last_fullsized":"2024-04-20T02:53:31.148891+0000","mapping_epoch":154,"log_start":"153'40","ondisk_log_start":"153'40","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'36","last_scrub_stamp":"2024-04-20T02:52:46.138400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:46.138400+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":40,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:09.709596+0000","scrub_duration":0.003388247,"objects_trimmed":0,"snaptrim_duration":0.038600218999999998,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":116380,"num_write":27,"num_write_kb":7507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1714357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":125454988,"num_objects":51,"num_object_clones":10,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3101,"num_read_kb":3820415,"num_write":1090,"num_write_kb":300990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":222547162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":17,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":307884032,"data_stored":340189153,"data_compressed":515252,"data_compressed_allocated":434176,"data_compressed_original":1310720,"omap_allocated":2595,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":48,"acting":48,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":880,"num_read_kb":34624,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":66983468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":225505280,"data_stored":240729994,"data_compressed":12805896,"data_compressed_allocated":13840384,"data_compressed_original":29753344,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509485,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76872,"kb_used_data":41660,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94294968,"statfs":{"total":96636764160,"available":96558047232,"internally_reserved":0,"allocated":42659840,"data_stored":45652264,"data_compressed":90764,"data_compressed_allocated":102400,"data_compressed_original":774144,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.47199999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.501}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.52800000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.38100000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.435,"15min":0.443},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65500000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.442},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.59399999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.623}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.441,"15min":0.45000000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57699999999999996,"5min":2.431,"15min":2.431},"last":0.48399999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.57299999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103376,"kb_used_data":67460,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94268464,"statfs":{"total":96636764160,"available":96530907136,"internally_reserved":0,"allocated":69079040,"data_stored":72476477,"data_compressed":1987361,"data_compressed_allocated":2129920,"data_compressed_original":4653056,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56599999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.68700000000000006}]},{"osd":1,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42299999999999999,"15min":0.41699999999999998},"min":{"1min":0.34699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.64500000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.47899999999999998,"15min":0.46100000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.437,"15min":0.434},"min":{"1min":0.34499999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.48399999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.625}]},{"osd":6,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.27900000000000003,"5min":0.252,"15min":0.252},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.432},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.441,"15min":0.437},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.56399999999999995}]}]},{"osd":2,"up_from":0,"seq":592705486861,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120660,"kb_used_data":85444,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94251180,"statfs":{"total":96636764160,"available":96513208320,"internally_reserved":0,"allocated":87494656,"data_stored":93957482,"data_compressed":1830696,"data_compressed_allocated":1974272,"data_compressed_original":4153344,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.311}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.378}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.753,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.63600000000000001,"5min":1.097,"15min":1.097},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.53500000000000003,"15min":0.54900000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.56399999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.436}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.5},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.57099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108512,"kb_used_data":73292,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94263328,"statfs":{"total":96636764160,"available":96525647872,"internally_reserved":0,"allocated":75051008,"data_stored":80592938,"data_compressed":1956129,"data_compressed_allocated":2117632,"data_compressed_original":4587520,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.35099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.372}]},{"osd":2,"last update":"Sat Apr 20 02:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":1.198,"15min":1.198},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.33300000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.85899999999999999,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.52900000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.01,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.67600000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47199999999999998,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69899999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.66500000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60099999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":130572,"kb_used_data":95352,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94241268,"statfs":{"total":96636764160,"available":96503058432,"internally_reserved":0,"allocated":97640448,"data_stored":104341490,"data_compressed":2785685,"data_compressed_allocated":3022848,"data_compressed_original":7024640,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.436},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.495}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.372},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.48199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.52400000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.625,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.55700000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.57799999999999996,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.66100000000000003,"5min":1.038,"15min":1.038},"last":0.56699999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.58999999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119488,"kb_used_data":84264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94252352,"statfs":{"total":96636764160,"available":96514408448,"internally_reserved":0,"allocated":86286336,"data_stored":89694042,"data_compressed":2819490,"data_compressed_allocated":3063808,"data_compressed_original":6762496,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.379}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.44900000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.36599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90500000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.66400000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69599999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.61099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.58299999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.68400000000000005}]}]},{"osd":4,"up_from":0,"seq":717259538439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49860608,"data_stored":52937332,"data_compressed":2759802,"data_compressed_allocated":2973696,"data_compressed_original":6184960,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31961088,"data_stored":33949988,"data_compressed":1733638,"data_compressed_allocated":1871872,"data_compressed_original":3956736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12894208,"data_stored":13909415,"data_compressed":734140,"data_compressed_allocated":790528,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18509824,"data_stored":19818665,"data_compressed":1037033,"data_compressed_allocated":1118208,"data_compressed_original":2482176,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34250752,"data_stored":35250075,"data_compressed":54370,"data_compressed_allocated":57344,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53436416,"data_stored":59040834,"data_compressed":97058,"data_compressed_allocated":102400,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2595,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40202240,"data_stored":44569875,"data_compressed":11526,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56590336,"data_stored":67896718,"data_compressed":95804,"data_compressed_allocated":53248,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":34230272,"data_stored":36453414,"data_compressed":42758,"data_compressed_allocated":24576,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":48689152,"data_stored":52832608,"data_compressed":128290,"data_compressed_allocated":114688,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40484864,"data_stored":44145629,"data_compressed":85446,"data_compressed_allocated":69632,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:53:36.693 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-04-20T02:53:36.758 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:36.759 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:36.769 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:36.782 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:36.783+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:36.882 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:36.883+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:36.896 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:36.896 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:36.906 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:36.984 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:36.985+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:37.036 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:37.036 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:37.047 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:37.084 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:37.085+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:37.174 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:37.175 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:37.185 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:37.187 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:37.187+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:37.287 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:37.288+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:37.315 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:37.316 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:37.316 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:53:37.326 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:37.388 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:37.389+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:37.455 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:37.456 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:37.466 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:37.507 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:37.508+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:37.590 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:37.591+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:37.634 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:37.635 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:37.644 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:37.681 INFO:tasks.rados.rados.0.smithi105.stderr:0 errors. 2024-04-20T02:53:37.681 INFO:tasks.rados.rados.0.smithi105.stderr: 2024-04-20T02:53:37.684 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T02:53:37.725 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:37.725+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:37.772 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:37.773 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:37.783 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:37.827 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:37.827+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:37.894 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:37.895+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:37.920 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:37.920 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:37.931 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:38.015 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:38.015+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:38.070 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:53:38.070 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":197,"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","created":"2024-04-20T02:50:07.356902+0000","modified":"2024-04-20T02:53:37.305242+0000","last_up_change":"2024-04-20T02:53:06.173366+0000","last_in_change":"2024-04-20T02:52:50.391054+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T02:50:17.498569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99730110168457031,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-20T02:50:25.955448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99730110168457031,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-04-20T02:50:33.168857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.99730110168457031,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-04-20T02:50:33.714259+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"4.10","ready_epoch":193,"last_epoch_started":192,"last_epoch_clean":192,"source_version":"184'144","target_version":"186'205"},"last_change":"193","last_force_op_resend":"0","last_force_op_resend_prenautilus":"193","last_force_op_resend_preluminous":"191","auid":0,"snap_mode":"selfmanaged","snap_seq":99,"snap_epoch":187,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_acting":2.5099999904632568,"score_stable":2.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99730110168457031,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6","up":1,"in":1,"weight":0.8316650390625,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":43330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":43330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6815","nonce":43330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":43330}]},"public_addr":"172.21.15.105:6812/43330","cluster_addr":"172.21.15.105:6813/43330","heartbeat_back_addr":"172.21.15.105:6815/43330","heartbeat_front_addr":"172.21.15.105:6814/43330","state":["exists","up"]},{"osd":1,"uuid":"c5ae8a64-fc7c-47ab-af6a-4b762a840645","up":1,"in":1,"weight":0.8221893310546875,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6805","nonce":43331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6807","nonce":43331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43331}]},"public_addr":"172.21.15.105:6804/43331","cluster_addr":"172.21.15.105:6805/43331","heartbeat_back_addr":"172.21.15.105:6807/43331","heartbeat_front_addr":"172.21.15.105:6806/43331","state":["exists","up"]},{"osd":2,"uuid":"336e852e-5fa6-4ac0-b053-d31ad40d7c4e","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":121,"up_from":138,"up_thru":27,"down_at":122,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":66776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6801","nonce":66776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6803","nonce":66776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":66776}]},"public_addr":"172.21.15.105:6800/66776","cluster_addr":"172.21.15.105:6801/66776","heartbeat_back_addr":"172.21.15.105:6803/66776","heartbeat_front_addr":"172.21.15.105:6802/66776","state":["exists","up"]},{"osd":3,"uuid":"695cab3a-74d8-4046-ac61-2656dc0d893e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6809","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6811","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":43329}]},"public_addr":"172.21.15.105:6808/43329","cluster_addr":"172.21.15.105:6809/43329","heartbeat_back_addr":"172.21.15.105:6811/43329","heartbeat_front_addr":"172.21.15.105:6810/43329","state":["exists","up"]},{"osd":4,"uuid":"1d41da44-d894-40ac-acaa-aa0df15296c6","up":1,"in":0,"weight":0,"primary_affinity":0.9784088134765625,"last_clean_begin":15,"last_clean_end":149,"up_from":167,"up_thru":145,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":66272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":66272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":66272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":66272}]},"public_addr":"172.21.15.138:6808/66272","cluster_addr":"172.21.15.138:6809/66272","heartbeat_back_addr":"172.21.15.138:6811/66272","heartbeat_front_addr":"172.21.15.138:6810/66272","state":["exists","up"]},{"osd":5,"uuid":"3e1c33fa-6c8c-4898-99bc-462ecef25f30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":25,"up_from":38,"up_thru":189,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":52590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":52590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":52590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":52590}]},"public_addr":"172.21.15.138:6802/52590","cluster_addr":"172.21.15.138:6804/52590","heartbeat_back_addr":"172.21.15.138:6807/52590","heartbeat_front_addr":"172.21.15.138:6806/52590","state":["exists","up"]},{"osd":6,"uuid":"fd1fc124-7cf8-4628-9076-1f27d6385676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":43329}]},"public_addr":"172.21.15.138:6800/43329","cluster_addr":"172.21.15.138:6801/43329","heartbeat_back_addr":"172.21.15.138:6805/43329","heartbeat_front_addr":"172.21.15.138:6803/43329","state":["exists","up"]},{"osd":7,"uuid":"53ae8193-5c3e-4698-aacd-df2ad1db9143","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":43351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6813","nonce":43351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6815","nonce":43351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":43351}]},"public_addr":"172.21.15.138:6812/43351","cluster_addr":"172.21.15.138:6813/43351","heartbeat_back_addr":"172.21.15.138:6815/43351","heartbeat_front_addr":"172.21.15.138:6814/43351","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:52:44.296659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:52:23.214623+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-20T02:52:56.806719+0000","dead_epoch":122},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:52:58.973108+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":49625,"last_purged_snaps_scrub":"2024-04-20T02:52:38.383284+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:52:29.483129+0000","dead_epoch":26},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:52:18.346058+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:32.093504+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.0","osds":[6,0,2]}],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":0,"to":7}]},{"pgid":"4.b","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":78,"length":1},{"begin":80,"length":1},{"begin":83,"length":3},{"begin":90,"length":1},{"begin":92,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T02:53:38.085 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:53:38.098 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:38.099+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:38.102 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:38.102 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:38.113 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:38.200 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:38.200+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:38.286 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:38.286 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:38.297 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:38.300 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:38.300+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:38.401 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:38.401+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:38.437 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:53:38.438 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:53:38.438 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:38.439 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:38.449 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:38.450 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-04-20T02:53:37.481872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":224121424,"num_objects":181,"num_object_clones":10,"num_object_copies":543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":4097,"num_read_kb":3855139,"num_write":2470,"num_write_kb":627372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":289530630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":17,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":659480,"kb_used_data":447472,"kb_used_omap":9,"kb_used_meta":211894,"kb_avail":565571560,"statfs":{"total":579820584960,"available":579145277440,"internally_reserved":0,"allocated":458211328,"data_stored":486714693,"data_compressed":11470125,"data_compressed_allocated":12410880,"data_compressed_original":27955200,"omap_allocated":9619,"internal_metadata":216980077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.326375"},"pg_stats":[{"pgid":"4.1","version":"184'139","reported_seq":703,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.156534+0000","last_change":"2024-04-20T02:53:29.153095+0000","last_active":"2024-04-20T02:53:32.156534+0000","last_peered":"2024-04-20T02:53:32.156534+0000","last_clean":"2024-04-20T02:53:32.156534+0000","last_became_active":"2024-04-20T02:53:29.152688+0000","last_became_peered":"2024-04-20T02:53:29.152688+0000","last_unstale":"2024-04-20T02:53:32.156534+0000","last_undegraded":"2024-04-20T02:53:32.156534+0000","last_fullsized":"2024-04-20T02:53:32.156534+0000","mapping_epoch":154,"log_start":"184'139","ondisk_log_start":"184'139","created":22,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'131","last_scrub_stamp":"2024-04-20T02:52:54.612743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.612743+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":368,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:59.591499+0000","scrub_duration":0.0022862500000000001,"objects_trimmed":1,"snaptrim_duration":0.011020622000000001,"stat_sum":{"num_bytes":8084332,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":252,"num_read_kb":318196,"num_write":63,"num_write_kb":15810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":28106829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":148,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.194997+0000","last_change":"2024-04-20T02:53:30.912852+0000","last_active":"2024-04-20T02:53:31.194997+0000","last_peered":"2024-04-20T02:53:31.194997+0000","last_clean":"2024-04-20T02:53:31.194997+0000","last_became_active":"2024-04-20T02:53:18.152172+0000","last_became_peered":"2024-04-20T02:53:18.152172+0000","last_unstale":"2024-04-20T02:53:31.194997+0000","last_undegraded":"2024-04-20T02:53:31.194997+0000","last_fullsized":"2024-04-20T02:53:31.194997+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:32.689737+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"186'205","reported_seq":786,"reported_epoch":193,"state":"peering","last_fresh":"2024-04-20T02:53:32.156419+0000","last_change":"2024-04-20T02:53:32.156622+0000","last_active":"2024-04-20T02:53:30.295880+0000","last_peered":"2024-04-20T02:53:31.155330+0000","last_clean":"2024-04-20T02:53:31.155330+0000","last_became_active":"2024-04-20T02:53:16.152778+0000","last_became_peered":"2024-04-20T02:53:31.151765+0000","last_unstale":"2024-04-20T02:53:32.156419+0000","last_undegraded":"2024-04-20T02:53:32.156419+0000","last_fullsized":"2024-04-20T02:53:32.156419+0000","mapping_epoch":176,"log_start":"186'205","ondisk_log_start":"186'205","created":22,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'67","last_scrub_stamp":"2024-04-20T02:51:19.252971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:19.252971+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":343,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001820667,"objects_trimmed":0,"snaptrim_duration":0.042836615000000001,"stat_sum":{"num_bytes":9545632,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":378,"num_read_kb":480412,"num_write":113,"num_write_kb":33005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33476197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":217,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.148717+0000","last_change":"2024-04-20T02:53:19.107136+0000","last_active":"2024-04-20T02:53:31.148717+0000","last_peered":"2024-04-20T02:53:31.148717+0000","last_clean":"2024-04-20T02:53:31.148717+0000","last_became_active":"2024-04-20T02:52:51.153987+0000","last_became_peered":"2024-04-20T02:52:51.153987+0000","last_unstale":"2024-04-20T02:53:31.148717+0000","last_undegraded":"2024-04-20T02:53:31.148717+0000","last_fullsized":"2024-04-20T02:53:31.148717+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:24.234001+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":218,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-20T02:53:33.158608+0000","last_change":"2024-04-20T02:52:53.937048+0000","last_active":"2024-04-20T02:53:33.158608+0000","last_peered":"2024-04-20T02:53:33.158608+0000","last_clean":"2024-04-20T02:53:33.158608+0000","last_became_active":"2024-04-20T02:52:51.154174+0000","last_became_peered":"2024-04-20T02:52:51.154174+0000","last_unstale":"2024-04-20T02:53:33.158608+0000","last_undegraded":"2024-04-20T02:53:33.158608+0000","last_fullsized":"2024-04-20T02:53:33.158608+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:53.936957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:53.936957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.656218+0000","scrub_duration":0.0013534090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"187'315","reported_seq":862,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.849744+0000","last_change":"2024-04-20T02:53:23.159920+0000","last_active":"2024-04-20T02:53:32.849744+0000","last_peered":"2024-04-20T02:53:32.849744+0000","last_clean":"2024-04-20T02:53:32.849744+0000","last_became_active":"2024-04-20T02:53:21.153338+0000","last_became_peered":"2024-04-20T02:53:21.153338+0000","last_unstale":"2024-04-20T02:53:32.849744+0000","last_undegraded":"2024-04-20T02:53:32.849744+0000","last_fullsized":"2024-04-20T02:53:32.849744+0000","mapping_epoch":47,"log_start":"186'312","ondisk_log_start":"186'312","created":22,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'243","last_scrub_stamp":"2024-04-20T02:52:54.597931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.597931+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":557,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:59.861555+0000","scrub_duration":0.0028458709999999998,"objects_trimmed":2,"snaptrim_duration":0.0095485269999999994,"stat_sum":{"num_bytes":15467220,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":413,"num_read_kb":563626,"num_write":89,"num_write_kb":25274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":19243742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.1","version":"187'261","reported_seq":601,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.849689+0000","last_change":"2024-04-20T02:53:08.766997+0000","last_active":"2024-04-20T02:53:32.849689+0000","last_peered":"2024-04-20T02:53:32.849689+0000","last_clean":"2024-04-20T02:53:32.849689+0000","last_became_active":"2024-04-20T02:52:52.148432+0000","last_became_peered":"2024-04-20T02:52:52.148432+0000","last_unstale":"2024-04-20T02:53:32.849689+0000","last_undegraded":"2024-04-20T02:53:32.849689+0000","last_fullsized":"2024-04-20T02:53:32.849689+0000","mapping_epoch":152,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:29.132762+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":233,"num_read_kb":8230,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":93,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-20T02:53:33.158259+0000","last_change":"2024-04-20T02:52:32.918930+0000","last_active":"2024-04-20T02:53:33.158259+0000","last_peered":"2024-04-20T02:53:33.158259+0000","last_clean":"2024-04-20T02:53:33.158259+0000","last_became_active":"2024-04-20T02:52:22.261878+0000","last_became_peered":"2024-04-20T02:52:22.261878+0000","last_unstale":"2024-04-20T02:53:33.158259+0000","last_undegraded":"2024-04-20T02:53:33.158259+0000","last_fullsized":"2024-04-20T02:53:33.158259+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:32.918813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.918813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:36.236915+0000","scrub_duration":0.0014288059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"186'153","reported_seq":733,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.855215+0000","last_change":"2024-04-20T02:53:31.855215+0000","last_active":"2024-04-20T02:53:31.855215+0000","last_peered":"2024-04-20T02:53:31.855215+0000","last_clean":"2024-04-20T02:53:31.855215+0000","last_became_active":"2024-04-20T02:53:16.151498+0000","last_became_peered":"2024-04-20T02:53:16.151498+0000","last_unstale":"2024-04-20T02:53:31.855215+0000","last_undegraded":"2024-04-20T02:53:31.855215+0000","last_fullsized":"2024-04-20T02:53:31.855215+0000","mapping_epoch":176,"log_start":"185'149","ondisk_log_start":"185'149","created":22,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'153","last_scrub_stamp":"2024-04-20T02:53:31.855077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:31.855077+0000","objects_scrubbed":5,"log_size":4,"log_dups_size":236,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.406736+0000","scrub_duration":0.0052908479999999999,"objects_trimmed":0,"snaptrim_duration":0.041643649999999997,"stat_sum":{"num_bytes":9717018,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":174,"num_read_kb":188998,"num_write":46,"num_write_kb":11974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22843153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.2","version":"192'274","reported_seq":592,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.197190+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:53:31.197190+0000","last_peered":"2024-04-20T02:53:31.197190+0000","last_clean":"2024-04-20T02:53:31.197190+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:53:31.197190+0000","last_undegraded":"2024-04-20T02:53:31.197190+0000","last_fullsized":"2024-04-20T02:53:31.197190+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":243,"num_read_kb":13907,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":361,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-20T02:53:33.158612+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:53:33.158612+0000","last_peered":"2024-04-20T02:53:33.158612+0000","last_clean":"2024-04-20T02:53:33.158612+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:53:33.158612+0000","last_undegraded":"2024-04-20T02:53:33.158612+0000","last_fullsized":"2024-04-20T02:53:33.158612+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":194,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.156615+0000","last_change":"2024-04-20T02:53:08.364116+0000","last_active":"2024-04-20T02:53:32.156615+0000","last_peered":"2024-04-20T02:53:32.156615+0000","last_clean":"2024-04-20T02:53:32.156615+0000","last_became_active":"2024-04-20T02:52:22.265487+0000","last_became_peered":"2024-04-20T02:52:22.265487+0000","last_unstale":"2024-04-20T02:53:32.156615+0000","last_undegraded":"2024-04-20T02:53:32.156615+0000","last_fullsized":"2024-04-20T02:53:32.156615+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.792324+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"186'105","reported_seq":362,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.194830+0000","last_change":"2024-04-20T02:53:25.163140+0000","last_active":"2024-04-20T02:53:31.194830+0000","last_peered":"2024-04-20T02:53:31.194830+0000","last_clean":"2024-04-20T02:53:31.194830+0000","last_became_active":"2024-04-20T02:53:18.152319+0000","last_became_peered":"2024-04-20T02:53:18.152319+0000","last_unstale":"2024-04-20T02:53:31.194830+0000","last_undegraded":"2024-04-20T02:53:31.194830+0000","last_fullsized":"2024-04-20T02:53:31.194830+0000","mapping_epoch":178,"log_start":"184'101","ondisk_log_start":"184'101","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'86","last_scrub_stamp":"2024-04-20T02:52:32.901998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.901998+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":131,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:43.974508+0000","scrub_duration":0.0024332220000000001,"objects_trimmed":0,"snaptrim_duration":0.99220328899999999,"stat_sum":{"num_bytes":5395179,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":35504,"num_write":16,"num_write_kb":3789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1543958,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.0","version":"190'241","reported_seq":526,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.148688+0000","last_change":"2024-04-20T02:52:42.201016+0000","last_active":"2024-04-20T02:53:31.148688+0000","last_peered":"2024-04-20T02:53:31.148688+0000","last_clean":"2024-04-20T02:53:31.148688+0000","last_became_active":"2024-04-20T02:52:25.272853+0000","last_became_peered":"2024-04-20T02:52:25.272853+0000","last_unstale":"2024-04-20T02:53:31.148688+0000","last_undegraded":"2024-04-20T02:53:31.148688+0000","last_fullsized":"2024-04-20T02:53:31.148688+0000","mapping_epoch":126,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'188","last_scrub_stamp":"2024-04-20T02:52:42.200865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:52:42.200865+0000","objects_scrubbed":31,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.320740+0000","scrub_duration":0.0052191850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":201,"num_read_kb":5875,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":15696807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":95,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.145249+0000","last_change":"2024-04-20T02:52:56.212555+0000","last_active":"2024-04-20T02:53:31.145249+0000","last_peered":"2024-04-20T02:53:31.145249+0000","last_clean":"2024-04-20T02:53:31.145249+0000","last_became_active":"2024-04-20T02:52:22.272822+0000","last_became_peered":"2024-04-20T02:52:22.272822+0000","last_unstale":"2024-04-20T02:53:31.145249+0000","last_undegraded":"2024-04-20T02:53:31.145249+0000","last_fullsized":"2024-04-20T02:53:31.145249+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:56.212430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.212430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:20.820102+0000","scrub_duration":0.0013799870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"184'130","reported_seq":720,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.156629+0000","last_change":"2024-04-20T02:53:23.161348+0000","last_active":"2024-04-20T02:53:32.156629+0000","last_peered":"2024-04-20T02:53:32.156629+0000","last_clean":"2024-04-20T02:53:32.156629+0000","last_became_active":"2024-04-20T02:52:46.141983+0000","last_became_peered":"2024-04-20T02:52:46.141983+0000","last_unstale":"2024-04-20T02:53:32.156629+0000","last_undegraded":"2024-04-20T02:53:32.156629+0000","last_fullsized":"2024-04-20T02:53:32.156629+0000","mapping_epoch":147,"log_start":"179'124","ondisk_log_start":"179'124","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'119","last_scrub_stamp":"2024-04-20T02:52:56.564632+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.564632+0000","objects_scrubbed":4,"log_size":6,"log_dups_size":310,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:22.552034+0000","scrub_duration":0.0028929759999999998,"objects_trimmed":1,"snaptrim_duration":0.010818365,"stat_sum":{"num_bytes":2840449,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":43301,"num_write":4,"num_write_kb":1274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"3.3","version":"184'235","reported_seq":540,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.849639+0000","last_change":"2024-04-20T02:53:18.737801+0000","last_active":"2024-04-20T02:53:32.849639+0000","last_peered":"2024-04-20T02:53:32.849639+0000","last_clean":"2024-04-20T02:53:32.849639+0000","last_became_active":"2024-04-20T02:53:11.150822+0000","last_became_peered":"2024-04-20T02:53:11.150822+0000","last_unstale":"2024-04-20T02:53:32.849639+0000","last_undegraded":"2024-04-20T02:53:32.849639+0000","last_fullsized":"2024-04-20T02:53:32.849639+0000","mapping_epoch":171,"log_start":"175'232","ondisk_log_start":"175'232","created":21,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":238,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.609708+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":203,"num_read_kb":6612,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":18551655,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":184,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.389983+0000","last_change":"2024-04-20T02:53:32.389983+0000","last_active":"2024-04-20T02:53:32.389983+0000","last_peered":"2024-04-20T02:53:32.389983+0000","last_clean":"2024-04-20T02:53:32.389983+0000","last_became_active":"2024-04-20T02:51:01.133709+0000","last_became_peered":"2024-04-20T02:51:01.133709+0000","last_unstale":"2024-04-20T02:53:32.389983+0000","last_undegraded":"2024-04-20T02:53:32.389983+0000","last_fullsized":"2024-04-20T02:53:32.389983+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:54.846797+0000","scrub_duration":0.0028872630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"185'48","reported_seq":508,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.148627+0000","last_change":"2024-04-20T02:53:25.199567+0000","last_active":"2024-04-20T02:53:31.148627+0000","last_peered":"2024-04-20T02:53:31.148627+0000","last_clean":"2024-04-20T02:53:31.148627+0000","last_became_active":"2024-04-20T02:53:17.156757+0000","last_became_peered":"2024-04-20T02:53:17.156757+0000","last_unstale":"2024-04-20T02:53:31.148627+0000","last_undegraded":"2024-04-20T02:53:31.148627+0000","last_fullsized":"2024-04-20T02:53:31.148627+0000","mapping_epoch":141,"log_start":"169'45","ondisk_log_start":"169'45","created":22,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'17","last_scrub_stamp":"2024-04-20T02:52:56.000178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.000178+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015078069999999999,"objects_trimmed":0,"snaptrim_duration":0.039989539999999997,"stat_sum":{"num_bytes":2991712,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":26723,"num_write":19,"num_write_kb":5030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5278912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.2","version":"0'0","reported_seq":266,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.156556+0000","last_change":"2024-04-20T02:53:25.404436+0000","last_active":"2024-04-20T02:53:32.156556+0000","last_peered":"2024-04-20T02:53:32.156556+0000","last_clean":"2024-04-20T02:53:32.156556+0000","last_became_active":"2024-04-20T02:53:25.402753+0000","last_became_peered":"2024-04-20T02:53:25.402753+0000","last_unstale":"2024-04-20T02:53:32.156556+0000","last_undegraded":"2024-04-20T02:53:32.156556+0000","last_fullsized":"2024-04-20T02:53:32.156556+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_clean_scrub_stamp":"2024-04-20T02:52:43.686760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.573670+0000","scrub_duration":0.0018670150000000001,"objects_trimmed":0,"snaptrim_duration":0.001125371,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":197,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.148458+0000","last_change":"2024-04-20T02:52:18.363613+0000","last_active":"2024-04-20T02:53:31.148458+0000","last_peered":"2024-04-20T02:53:31.148458+0000","last_clean":"2024-04-20T02:53:31.148458+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:53:31.148458+0000","last_undegraded":"2024-04-20T02:53:31.148458+0000","last_fullsized":"2024-04-20T02:53:31.148458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:18.363433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:18.363433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:38.745528+0000","scrub_duration":0.0016555929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"186'174","reported_seq":679,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-20T02:53:33.349960+0000","last_change":"2024-04-20T02:53:23.150954+0000","last_active":"2024-04-20T02:53:33.349960+0000","last_peered":"2024-04-20T02:53:33.349960+0000","last_clean":"2024-04-20T02:53:33.349960+0000","last_became_active":"2024-04-20T02:52:13.145577+0000","last_became_peered":"2024-04-20T02:52:13.145577+0000","last_unstale":"2024-04-20T02:53:33.349960+0000","last_undegraded":"2024-04-20T02:53:33.349960+0000","last_fullsized":"2024-04-20T02:53:33.349960+0000","mapping_epoch":62,"log_start":"186'170","ondisk_log_start":"186'170","created":22,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'74","last_scrub_stamp":"2024-04-20T02:51:56.784299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:56.784299+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":328,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015920019999999999,"objects_trimmed":0,"snaptrim_duration":0.00066820099999999999,"stat_sum":{"num_bytes":10729969,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":90,"num_read_kb":103553,"num_write":46,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.9","version":"192'507","reported_seq":1176,"reported_epoch":192,"state":"active+clean+snaptrim","last_fresh":"2024-04-20T02:53:31.194099+0000","last_change":"2024-04-20T02:53:31.192280+0000","last_active":"2024-04-20T02:53:31.194099+0000","last_peered":"2024-04-20T02:53:31.194099+0000","last_clean":"2024-04-20T02:53:31.194099+0000","last_became_active":"2024-04-20T02:53:11.151224+0000","last_became_peered":"2024-04-20T02:53:11.151224+0000","last_unstale":"2024-04-20T02:53:31.194099+0000","last_undegraded":"2024-04-20T02:53:31.194099+0000","last_fullsized":"2024-04-20T02:53:31.194099+0000","mapping_epoch":171,"log_start":"188'503","ondisk_log_start":"188'503","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'294","last_scrub_stamp":"2024-04-20T02:52:16.813424+0000","last_deep_scrub":"98'280","last_deep_scrub_stamp":"2024-04-20T02:51:56.947305+0000","last_clean_scrub_stamp":"2024-04-20T02:52:16.813424+0000","objects_scrubbed":7,"log_size":4,"log_dups_size":1124,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:41.845382+0000","scrub_duration":0.0034018329999999999,"objects_trimmed":0,"snaptrim_duration":0.041609871,"stat_sum":{"num_bytes":23290123,"num_objects":10,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":223,"num_read_kb":242421,"num_write":112,"num_write_kb":32076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":34386920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4d","length":"1"}]},{"pgid":"4.a","version":"184'120","reported_seq":593,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.194862+0000","last_change":"2024-04-20T02:53:23.158044+0000","last_active":"2024-04-20T02:53:31.194862+0000","last_peered":"2024-04-20T02:53:31.194862+0000","last_clean":"2024-04-20T02:53:31.194862+0000","last_became_active":"2024-04-20T02:53:18.153426+0000","last_became_peered":"2024-04-20T02:53:18.153426+0000","last_unstale":"2024-04-20T02:53:31.194862+0000","last_undegraded":"2024-04-20T02:53:31.194862+0000","last_fullsized":"2024-04-20T02:53:31.194862+0000","mapping_epoch":178,"log_start":"177'116","ondisk_log_start":"177'116","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'53","last_scrub_stamp":"2024-04-20T02:52:05.810627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:05.810627+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":116,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022334680000000002,"objects_trimmed":0,"snaptrim_duration":0.00042807000000000001,"stat_sum":{"num_bytes":6343443,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":204,"num_read_kb":230945,"num_write":64,"num_write_kb":16950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.b","version":"184'350","reported_seq":923,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-20T02:53:32.849671+0000","last_change":"2024-04-20T02:53:30.856293+0000","last_active":"2024-04-20T02:53:32.849671+0000","last_peered":"2024-04-20T02:53:32.849671+0000","last_clean":"2024-04-20T02:53:32.849671+0000","last_became_active":"2024-04-20T02:53:11.147630+0000","last_became_peered":"2024-04-20T02:53:11.147630+0000","last_unstale":"2024-04-20T02:53:32.849671+0000","last_undegraded":"2024-04-20T02:53:32.849671+0000","last_fullsized":"2024-04-20T02:53:32.849671+0000","mapping_epoch":171,"log_start":"179'344","ondisk_log_start":"179'344","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'350","last_scrub_stamp":"2024-04-20T02:53:30.856060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.856060+0000","objects_scrubbed":5,"log_size":6,"log_dups_size":344,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.470953+0000","scrub_duration":0.0031057490000000001,"objects_trimmed":1,"snaptrim_duration":0.0080676189999999998,"stat_sum":{"num_bytes":11923149,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":388,"num_read_kb":461817,"num_write":178,"num_write_kb":48621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":23150965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.c","version":"178'159","reported_seq":646,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.145477+0000","last_change":"2024-04-20T02:53:23.167866+0000","last_active":"2024-04-20T02:53:31.145477+0000","last_peered":"2024-04-20T02:53:31.145477+0000","last_clean":"2024-04-20T02:53:31.145477+0000","last_became_active":"2024-04-20T02:53:11.148476+0000","last_became_peered":"2024-04-20T02:53:11.148476+0000","last_unstale":"2024-04-20T02:53:31.145477+0000","last_undegraded":"2024-04-20T02:53:31.145477+0000","last_fullsized":"2024-04-20T02:53:31.145477+0000","mapping_epoch":171,"log_start":"173'155","ondisk_log_start":"173'155","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'95","last_scrub_stamp":"2024-04-20T02:52:09.803923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:09.803923+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":155,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:32.262327+0000","scrub_duration":0.0026115750000000001,"objects_trimmed":0,"snaptrim_duration":0.010578568,"stat_sum":{"num_bytes":4773320,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":178,"num_read_kb":193067,"num_write":90,"num_write_kb":24779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":14051707,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.d","version":"188'288","reported_seq":897,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.151054+0000","last_change":"2024-04-20T02:53:25.198992+0000","last_active":"2024-04-20T02:53:31.151054+0000","last_peered":"2024-04-20T02:53:31.151054+0000","last_clean":"2024-04-20T02:53:31.151054+0000","last_became_active":"2024-04-20T02:52:46.144032+0000","last_became_peered":"2024-04-20T02:52:46.144032+0000","last_unstale":"2024-04-20T02:53:31.151054+0000","last_undegraded":"2024-04-20T02:53:31.151054+0000","last_fullsized":"2024-04-20T02:53:31.151054+0000","mapping_epoch":147,"log_start":"186'285","ondisk_log_start":"186'285","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'169","last_scrub_stamp":"2024-04-20T02:52:08.841360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:08.841360+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":285,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002314502,"objects_trimmed":0,"snaptrim_duration":0.039094669999999998,"stat_sum":{"num_bytes":8353244,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":439,"num_read_kb":540894,"num_write":164,"num_write_kb":44217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":27793283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.e","version":"184'124","reported_seq":638,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.194854+0000","last_change":"2024-04-20T02:53:25.168677+0000","last_active":"2024-04-20T02:53:31.194854+0000","last_peered":"2024-04-20T02:53:31.194854+0000","last_clean":"2024-04-20T02:53:31.194854+0000","last_became_active":"2024-04-20T02:53:18.149105+0000","last_became_peered":"2024-04-20T02:53:18.149105+0000","last_unstale":"2024-04-20T02:53:31.194854+0000","last_undegraded":"2024-04-20T02:53:31.194854+0000","last_fullsized":"2024-04-20T02:53:31.194854+0000","mapping_epoch":178,"log_start":"173'118","ondisk_log_start":"173'118","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'77","last_scrub_stamp":"2024-04-20T02:51:46.088382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:51:46.088382+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":118,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002170693,"objects_trimmed":0,"snaptrim_duration":0.99770691499999997,"stat_sum":{"num_bytes":3882450,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":196,"num_read_kb":274578,"num_write":59,"num_write_kb":18271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":10957139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]},{"pgid":"4.f","version":"163'44","reported_seq":493,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-04-20T02:53:31.148891+0000","last_change":"2024-04-20T02:53:25.198652+0000","last_active":"2024-04-20T02:53:31.148891+0000","last_peered":"2024-04-20T02:53:31.148891+0000","last_clean":"2024-04-20T02:53:31.148891+0000","last_became_active":"2024-04-20T02:52:54.156070+0000","last_became_peered":"2024-04-20T02:52:54.156070+0000","last_unstale":"2024-04-20T02:53:31.148891+0000","last_undegraded":"2024-04-20T02:53:31.148891+0000","last_fullsized":"2024-04-20T02:53:31.148891+0000","mapping_epoch":154,"log_start":"153'40","ondisk_log_start":"153'40","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'36","last_scrub_stamp":"2024-04-20T02:52:46.138400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:46.138400+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":40,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:09.709596+0000","scrub_duration":0.003388247,"objects_trimmed":0,"snaptrim_duration":0.038600218999999998,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":116380,"num_write":27,"num_write_kb":7507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1714357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4d","length":"2"},{"start":"50","length":"1"},{"start":"53","length":"3"},{"start":"5a","length":"1"},{"start":"5c","length":"2"}]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":125454988,"num_objects":51,"num_object_clones":10,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3101,"num_read_kb":3820415,"num_write":1090,"num_write_kb":300990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":222547162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":17,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":307884032,"data_stored":340189153,"data_compressed":515252,"data_compressed_allocated":434176,"data_compressed_original":1310720,"omap_allocated":2595,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":48,"acting":48,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":880,"num_read_kb":34624,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":66983468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":225505280,"data_stored":240729994,"data_compressed":12805896,"data_compressed_allocated":13840384,"data_compressed_original":29753344,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509485,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76872,"kb_used_data":41660,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94294968,"statfs":{"total":96636764160,"available":96558047232,"internally_reserved":0,"allocated":42659840,"data_stored":45652264,"data_compressed":90764,"data_compressed_allocated":102400,"data_compressed_original":774144,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.47199999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.501}]},{"osd":2,"last update":"Sat Apr 20 02:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.52800000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.38100000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.435,"15min":0.443},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65500000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.442},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.59399999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.623}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.441,"15min":0.45000000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57699999999999996,"5min":2.431,"15min":2.431},"last":0.48399999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.57299999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103376,"kb_used_data":67460,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94268464,"statfs":{"total":96636764160,"available":96530907136,"internally_reserved":0,"allocated":69079040,"data_stored":72476477,"data_compressed":1987361,"data_compressed_allocated":2129920,"data_compressed_original":4653056,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56599999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.68700000000000006}]},{"osd":1,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42299999999999999,"15min":0.41699999999999998},"min":{"1min":0.34699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.64500000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.47899999999999998,"15min":0.46100000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.437,"15min":0.434},"min":{"1min":0.34499999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.48399999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.625}]},{"osd":6,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.27900000000000003,"5min":0.252,"15min":0.252},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.432},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.441,"15min":0.437},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.56399999999999995}]}]},{"osd":2,"up_from":0,"seq":592705486861,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120660,"kb_used_data":85444,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94251180,"statfs":{"total":96636764160,"available":96513208320,"internally_reserved":0,"allocated":87494656,"data_stored":93957482,"data_compressed":1830696,"data_compressed_allocated":1974272,"data_compressed_original":4153344,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.311}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.378}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.753,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.63600000000000001,"5min":1.097,"15min":1.097},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.53500000000000003,"15min":0.54900000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.56399999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.436}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.5},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.57099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108512,"kb_used_data":73292,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94263328,"statfs":{"total":96636764160,"available":96525647872,"internally_reserved":0,"allocated":75051008,"data_stored":80592938,"data_compressed":1956129,"data_compressed_allocated":2117632,"data_compressed_original":4587520,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.35099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.372}]},{"osd":2,"last update":"Sat Apr 20 02:52:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.40300000000000002,"15min":0.41099999999999998},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":1.198,"15min":1.198},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.33300000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.85899999999999999,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.52900000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.01,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.67600000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47199999999999998,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69899999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.66500000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60099999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":130572,"kb_used_data":95352,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94241268,"statfs":{"total":96636764160,"available":96503058432,"internally_reserved":0,"allocated":97640448,"data_stored":104341490,"data_compressed":2785685,"data_compressed_allocated":3022848,"data_compressed_original":7024640,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.436},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.495}]},{"osd":2,"last update":"Sat Apr 20 02:51:16 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.372},{"interface":"front","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.0020000000000007,"5min":8.0020000000000007,"15min":8.0020000000000007},"last":0.48199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.52400000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.625,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.55700000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.57799999999999996,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.66100000000000003,"5min":1.038,"15min":1.038},"last":0.56699999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.58999999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119488,"kb_used_data":84264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94252352,"statfs":{"total":96636764160,"available":96514408448,"internally_reserved":0,"allocated":86286336,"data_stored":89694042,"data_compressed":2819490,"data_compressed_allocated":3063808,"data_compressed_original":6762496,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.379}]},{"osd":2,"last update":"Sat Apr 20 02:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.244,"5min":1.244,"15min":1.244},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.44900000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.36599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90500000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.66400000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69599999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.61099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.58299999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.68400000000000005}]}]},{"osd":4,"up_from":0,"seq":717259538439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49860608,"data_stored":52937332,"data_compressed":2759802,"data_compressed_allocated":2973696,"data_compressed_original":6184960,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31961088,"data_stored":33949988,"data_compressed":1733638,"data_compressed_allocated":1871872,"data_compressed_original":3956736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12894208,"data_stored":13909415,"data_compressed":734140,"data_compressed_allocated":790528,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18509824,"data_stored":19818665,"data_compressed":1037033,"data_compressed_allocated":1118208,"data_compressed_original":2482176,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34250752,"data_stored":35250075,"data_compressed":54370,"data_compressed_allocated":57344,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53436416,"data_stored":59040834,"data_compressed":97058,"data_compressed_allocated":102400,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2595,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40202240,"data_stored":44569875,"data_compressed":11526,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56590336,"data_stored":67896718,"data_compressed":95804,"data_compressed_allocated":53248,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":34230272,"data_stored":36453414,"data_compressed":42758,"data_compressed_allocated":24576,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":48689152,"data_stored":52832608,"data_compressed":128290,"data_compressed_allocated":114688,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40484864,"data_stored":44145629,"data_compressed":85446,"data_compressed_allocated":69632,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:53:38.453 INFO:tasks.ceph.ceph_manager.ceph:pg 4.9 in trimming, state: active+clean+snaptrim 2024-04-20T02:53:38.453 INFO:tasks.ceph.ceph_manager.ceph:unique_pool_0 still trimming, waiting 2024-04-20T02:53:38.502 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:38.502+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:38.581 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:38.581 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:38.591 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:38.603 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:38.604+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:38.615 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:38.704 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:38.704+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:38.760 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:38.761 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:38.761 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:38.771 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:38.779 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:38.800 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:38.805 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:38.805+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:38.838 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:38.876 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:38.906 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:38.907+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:38.911 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:38.933 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:38.943 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:38.943 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:38.943 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:38.943 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:38.953 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:38.954 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:38.955 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:38.955 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:38.955 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:38.955 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:38.955 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:38.955 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:38.955 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:38.955 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:39.008 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:39.008+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:39.108 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:39.109+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:39.134 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:39.134 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:39.134 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:39.134 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:39.134 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:39.145 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:39.209 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:39.210+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:39.275 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:39.275 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:39.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:39.275 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:39.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.275 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 168.341550571, 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:39.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:39.281 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 168.34129036499999, 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:53:39.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:53:39.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:53:39.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 168.34126889699999, 2024-04-20T02:53:39.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:53:39.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3771 4.b 4:d4308fef:::smithi10548823-6:head [write 2548663~763022] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.092240240000002, 2024-04-20T02:53:39.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072733780999999997, 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3771 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181891+0000", 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182301+0000", 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040985099999999998 2024-04-20T02:53:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182317+0000", 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6305999999999999e-05 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182329+0000", 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1337e-05 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198720+0000", 2024-04-20T02:53:39.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016391606 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198807+0000", 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7319999999999997e-05 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198925+0000", 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117345 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218062+0000", 2024-04-20T02:53:39.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019137134 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234990+0000", 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016928367 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254488+0000", 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019497460000000001 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254571+0000", 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3029999999999996e-05 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254627+0000", 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5865000000000001e-05 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3772 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3395047] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:39.298 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.091793668000001, 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072419708999999999, 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3772 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182343+0000", 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0099999999999997e-06 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182349+0000", 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9569999999999999e-06 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182354+0000", 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.826e-06 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217228+0000", 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034874555000000002 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217312+0000", 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3478000000000003e-05 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217431+0000", 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119512 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234969+0000", 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017537833999999999 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235012+0000", 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3411000000000002e-05 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254638+0000", 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019625815000000001 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254682+0000", 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3945e-05 2024-04-20T02:53:39.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254759+0000", 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6882e-05 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3773 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.091775061, 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072538680999999994, 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.304 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3773 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:39.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182360+0000", 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3309999999999998e-06 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182366+0000", 2024-04-20T02:53:39.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.981e-06 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182371+0000", 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5390000000000001e-06 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217932+0000", 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035561700000000002 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218020+0000", 2024-04-20T02:53:39.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7352e-05 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254778+0000", 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036757971 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254819+0000", 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1525e-05 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254897+0000", 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7621999999999996e-05 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4057 4.b 4:d4308fef:::smithi10548823-6:head [write 749678~790384,stat] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.130920320999998, 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074563737000000005, 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.309 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4057 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143210+0000", 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143760+0000", 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055015999999999995 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143772+0000", 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1345000000000001e-05 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143781+0000", 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0929999999999992e-06 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143895+0000", 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113998 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143906+0000", 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1195999999999999e-05 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.153590+0000", 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096836000000000005 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155611+0000", 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002021012 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217691+0000", 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062080244 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217777+0000", 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5583999999999999e-05 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4058 4.b 4:d4308fef:::smithi10548823-6:head [write 2332155~504118] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.130339294999999, 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074090323999999999, 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:39.314 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4058 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143793+0000", 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144104+0000", 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000310971 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144112+0000", 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1990000000000006e-06 2024-04-20T02:53:39.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144118+0000", 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.443e-06 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144162+0000", 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3066999999999999e-05 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144172+0000", 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0083e-05 2024-04-20T02:53:39.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155911+0000", 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011739657000000001 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155993+0000", 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1754000000000004e-05 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217810+0000", 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061816958999999998 2024-04-20T02:53:39.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217884+0000", 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4098999999999996e-05 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4059 4.b 4:d4308fef:::smithi10548823-6:head [write 3271857~549469] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.130003133999999, 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073860173000000001, 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.319 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4059 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:39.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144129+0000", 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144439+0000", 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030924000000000001 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144447+0000", 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8930000000000005e-06 2024-04-20T02:53:39.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144455+0000", 2024-04-20T02:53:39.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1410000000000005e-06 2024-04-20T02:53:39.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144505+0000", 2024-04-20T02:53:39.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0698000000000001e-05 2024-04-20T02:53:39.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144520+0000", 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4605e-05 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156034+0000", 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011513809 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.327 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156130+0000", 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6039999999999995e-05 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217915+0000", 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061785501999999999 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217990+0000", 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4770000000000004e-05 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.328 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4060 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3949678] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.129667755, 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085188106999999999, 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4060 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.329 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.330 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144469+0000", 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.946e-06 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144476+0000", 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7909999999999999e-06 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144482+0000", 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.601e-06 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.331 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144803+0000", 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032103699999999998 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144813+0000", 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0183999999999999e-05 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156168+0000", 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011354523 2024-04-20T02:53:39.332 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156254+0000", 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5706999999999998e-05 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218023+0000", 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061769566999999997 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218137+0000", 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113912 2024-04-20T02:53:39.333 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218377+0000", 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023941000000000001 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225107+0000", 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067304510000000001 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225121+0000", 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4341e-05 2024-04-20T02:53:39.334 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229481+0000", 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043599880000000004 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229522+0000", 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0224e-05 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229653+0000", 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013190500000000001 2024-04-20T02:53:39.335 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4061 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.129528137999998, 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085431572999999997, 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4061 2024-04-20T02:53:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144604+0000", 2024-04-20T02:53:39.337 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144608+0000", 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1080000000000003e-06 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144618+0000", 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5149999999999995e-06 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144625+0000", 2024-04-20T02:53:39.338 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8260000000000004e-06 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144846+0000", 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022160699999999999 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144854+0000", 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0399999999999993e-06 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156327+0000", 2024-04-20T02:53:39.339 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011472178 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156457+0000", 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013012599999999999 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218704+0000", 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062247080000000003 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218795+0000", 2024-04-20T02:53:39.340 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1391000000000006e-05 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229703+0000", 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010907834999999999 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229818+0000", 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115423 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.230037+0000", 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021803600000000001 2024-04-20T02:53:39.341 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4146 3.1 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head [write 0~1129472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.093932160999998, 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067860845000000003, 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.342 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4146 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.343 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180199+0000", 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180439+0000", 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000239375 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180449+0000", 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0220999999999999e-05 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.344 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180456+0000", 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0090000000000001e-06 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181170+0000", 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071358900000000004 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181331+0000", 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016178500000000001 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.345 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,5", 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181490+0000", 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158703 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225260+0000", 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043770166999999999 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226398+0000", 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001138189 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.346 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247971+0000", 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021572728999999999 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248021+0000", 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0176999999999997e-05 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248062+0000", 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0472000000000001e-05 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.347 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4623 4.3 4:c07c6daa:::smithi10548823-7:head [write 2772312~34523] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.124004313, 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074418926999999996, 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4623 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.348 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150128+0000", 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.349 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150143+0000", 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5166e-05 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150147+0000", 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.924e-06 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150151+0000", 2024-04-20T02:53:39.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.777e-06 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150180+0000", 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8472000000000001e-05 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150184+0000", 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4349999999999999e-06 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187063+0000", 2024-04-20T02:53:39.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036878474000000001 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187145+0000", 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2107999999999994e-05 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187280+0000", 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135384 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189223+0000", 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019430820000000001 2024-04-20T02:53:39.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189411+0000", 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000188061 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224429+0000", 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035017386999999997 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224487+0000", 2024-04-20T02:53:39.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8884000000000001e-05 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224548+0000", 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0238000000000002e-05 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4624 4.3 4:c07c6daa:::smithi10548823-7:head [setxattr _header (56) in=7b,truncate 2806835] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.123978068, 2024-04-20T02:53:39.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074609524999999996, 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4624 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150158+0000", 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9799999999999998e-06 2024-04-20T02:53:39.356 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150161+0000", 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8140000000000001e-06 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150164+0000", 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5329999999999999e-06 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150197+0000", 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2926999999999998e-05 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150201+0000", 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2989999999999998e-06 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188464+0000", 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038262276999999997 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188576+0000", 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112413 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188756+0000", 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000180206 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189459+0000", 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070307500000000005 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224557+0000", 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035097721999999998 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224633+0000", 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5798999999999997e-05 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224690+0000", 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6784999999999999e-05 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224765+0000", 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4969999999999995e-05 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4625 4.3 4:c07c6daa:::smithi10548823-7:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.123966106000001, 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074770387999999993, 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.361 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4625 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:39.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150169+0000", 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7680000000000001e-06 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150171+0000", 2024-04-20T02:53:39.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8779999999999998e-06 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150175+0000", 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2909999999999999e-06 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150215+0000", 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0074000000000003e-05 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150219+0000", 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.177e-06 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189474+0000", 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039255274999999999 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189531+0000", 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7188999999999997e-05 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224802+0000", 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035270981999999999 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224861+0000", 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8276999999999998e-05 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224937+0000", 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6689999999999999e-05 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.366 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4661 4.3 4:cd374330:::smithi10548823-1:head [write 490870~618799] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.123955347999999, 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074896037999999998, 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4661 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:39.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150452+0000", 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027429199999999998 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150458+0000", 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5810000000000002e-06 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150463+0000", 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7029999999999997e-06 2024-04-20T02:53:39.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150486+0000", 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3308000000000001e-05 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150491+0000", 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6179999999999997e-06 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189565+0000", 2024-04-20T02:53:39.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039073926000000002 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189713+0000", 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014765700000000001 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189930+0000", 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021749700000000001 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224591+0000", 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034661210999999997 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224775+0000", 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000184091 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224987+0000", 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021128100000000001 2024-04-20T02:53:39.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225040+0000", 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3395e-05 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225074+0000", 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3699999999999999e-05 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.373 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4662 4.3 4:cd374330:::smithi10548823-1:head [write 1592676~774144] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.123664464000001, 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075836991000000006, 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4662 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.374 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150468+0000", 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150808+0000", 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033972899999999999 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150815+0000", 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8159999999999996e-06 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150819+0000", 2024-04-20T02:53:39.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150855+0000", 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150886+0000", 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0876000000000001e-05 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.204837+0000", 2024-04-20T02:53:39.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053950873000000003 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.204916+0000", 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9008e-05 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.205039+0000", 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012294000000000001 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224780+0000", 2024-04-20T02:53:39.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019741771000000002 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225703+0000", 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00092290600000000003 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226196+0000", 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049284199999999995 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226250+0000", 2024-04-20T02:53:39.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4413000000000002e-05 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226306+0000", 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.516e-05 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4663 4.3 4:cd374330:::smithi10548823-1:head [write 2793074~97796] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.123308891000001, 2024-04-20T02:53:39.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075759200999999998, 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4663 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150883+0000", 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9308999999999997e-05 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150888+0000", 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1139999999999997e-06 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150892+0000", 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5899999999999999e-06 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150924+0000", 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2289000000000002e-05 2024-04-20T02:53:39.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150929+0000", 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0259999999999998e-06 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221580+0000", 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070650828999999998 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221708+0000", 2024-04-20T02:53:39.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012845700000000001 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221897+0000", 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000188789 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225650+0000", 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037528980000000002 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226379+0000", 2024-04-20T02:53:39.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072901200000000002 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226492+0000", 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112905 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226526+0000", 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711000000000003e-05 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226583+0000", 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7624999999999998e-05 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4664 4.3 4:cd374330:::smithi10548823-1:head [setxattr _header (56) in=7b,truncate 2890870] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.123235261000001, 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076283431999999998, 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.387 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4664 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150900+0000", 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.571e-06 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150903+0000", 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9409999999999999e-06 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.389 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150907+0000", 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5599999999999998e-06 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150988+0000", 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1544000000000004e-05 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151006+0000", 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7754999999999999e-05 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225114+0000", 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074107889999999996 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225189+0000", 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4851999999999994e-05 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225305+0000", 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011592899999999999 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225800+0000", 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049498200000000004 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226624+0000", 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00082437000000000005 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227055+0000", 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043119499999999999 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227113+0000", 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7500999999999997e-05 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227181+0000", 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8625000000000001e-05 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4665 4.3 4:cd374330:::smithi10548823-1:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:39.393 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 18.123210865000001, 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076413855000000003, 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4665 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:39.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150924+0000", 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8190000000000001e-06 2024-04-20T02:53:39.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150929+0000", 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6400000000000002e-06 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150933+0000", 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5039999999999998e-06 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151025+0000", 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2354000000000004e-05 2024-04-20T02:53:39.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151040+0000", 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708999999999999e-05 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225553+0000", 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074513356000000003 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225640+0000", 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6335000000000006e-05 2024-04-20T02:53:39.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227206+0000", 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001566448 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227261+0000", 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4907000000000001e-05 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227336+0000", 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5050999999999997e-05 2024-04-20T02:53:39.398 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.399 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.399 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.399 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.399 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.399 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:39.399 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:53:39.401 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:39.401 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:39.401+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:39.501 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:39.501+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:39.582 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:39.582 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:39.582 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:39.582 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:39.593 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:39.601 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:39.602+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:39.702 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:39.703+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:39.763 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:39.763 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:39.763 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:39.763 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:39.763 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:39.773 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:39.804 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:39.804+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:39.904 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:39.905+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:39.943 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 173.35745614499999, 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.944 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.945 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:53:39.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:53:39.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 173.33672683399999, 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.950 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.954 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 173.327475083, 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.956 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:53:39.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:53:39.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:53:39.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:53:39.961 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 165.53047586700001, 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:39.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:53:39.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:53:39.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:53:39.965 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 165.52447107899999, 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:39.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:53:39.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:53:39.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:53:39.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4029 4.13 4:cf11ccc0:::smithi10548823-4:head [write 699257~564637,stat] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.386928562999998, 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.86467550400000004, 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.970 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4029 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555666+0000", 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555809+0000", 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014306900000000001 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555832+0000", 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3680999999999999e-05 2024-04-20T02:53:39.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555846+0000", 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3654e-05 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555913+0000", 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6736999999999995e-05 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556058+0000", 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014538100000000001 2024-04-20T02:53:39.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556392+0000", 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033393099999999998 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578661+0000", 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022268553999999999 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578702+0000", 2024-04-20T02:53:39.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1533999999999999e-05 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578747+0000", 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4858999999999999e-05 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578811+0000", 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3520000000000002e-05 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605743+0000", 2024-04-20T02:53:39.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026932418999999999 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420312+0000", 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81456849600000003 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420343+0000", 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1911e-05 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4030 4.13 4:cf11ccc0:::smithi10548823-4:head [write 1873062~640438] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.380015442000001, 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85780235299999996, 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4030 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562579+0000", 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562714+0000", 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013475 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562728+0000", 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3516000000000001e-05 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562736+0000", 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5690000000000005e-06 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578835+0000", 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016099156 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578957+0000", 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121094 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.579123+0000", 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016647100000000001 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.980 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605727+0000", 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026603643999999999 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605749+0000", 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2070999999999999e-05 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605799+0000", 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0417000000000002e-05 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605841+0000", 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2344e-05 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605907+0000", 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5135999999999994e-05 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420373+0000", 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81446675899999998 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420383+0000", 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0108999999999999e-05 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4031 4.13 4:cf11ccc0:::smithi10548823-4:head [write 3305941~593316] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.372453866999997, 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85027709799999995, 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.983 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4031 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:39.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570142+0000", 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570277+0000", 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135606 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570288+0000", 2024-04-20T02:53:39.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0719e-05 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570294+0000", 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2519999999999996e-06 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592786+0000", 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022491220999999999 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592911+0000", 2024-04-20T02:53:39.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012546599999999999 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.593118+0000", 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000207011 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605878+0000", 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012760298999999999 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605934+0000", 2024-04-20T02:53:39.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5813999999999998e-05 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605963+0000", 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9275e-05 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606612+0000", 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00064813999999999996 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613782+0000", 2024-04-20T02:53:39.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071705670000000001 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420411+0000", 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.80662907399999995 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420420+0000", 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5790000000000004e-06 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.989 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4032 4.13 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (56) in=7b,truncate 3899257] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.372297439, 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85015831100000006, 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4032 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570302+0000", 2024-04-20T02:53:39.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1700000000000001e-06 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570305+0000", 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.976e-06 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570308+0000", 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8899999999999999e-06 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605985+0000", 2024-04-20T02:53:39.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035677121999999999 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606066+0000", 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0797000000000006e-05 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606211+0000", 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014569999999999999 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607013+0000", 2024-04-20T02:53:39.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00080160599999999995 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607046+0000", 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2938999999999998e-05 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607086+0000", 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0203000000000001e-05 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607157+0000", 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0621e-05 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613833+0000", 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066762269999999999 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420448+0000", 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.806615478 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420457+0000", 2024-04-20T02:53:39.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9779999999999994e-06 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4033 4.13 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.372285564999999, 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85021192700000003, 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:39.996 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4033 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570312+0000", 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419999999999999e-06 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570316+0000", 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3730000000000001e-06 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570318+0000", 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6680000000000001e-06 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606541+0000", 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036222797000000001 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606598+0000", 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6629999999999998e-05 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:39.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420515+0000", 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81391740099999998 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420523+0000", 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6799999999999993e-06 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4034 4.13 4:cfd1e4d4:::smithi10548823-9:head [rollback 3f] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:40.000 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.372275926999997, 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85017147900000001, 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4034 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570320+0000", 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570322+0000", 2024-04-20T02:53:40.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.37e-06 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570325+0000", 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7800000000000001e-06 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570328+0000", 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5379999999999998e-06 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606632+0000", 2024-04-20T02:53:40.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036304075999999998 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606716+0000", 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3689000000000005e-05 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420484+0000", 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81376812700000001 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420492+0000", 2024-04-20T02:53:40.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1179999999999992e-06 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4062 4.9 4:9c87d36d:::smithi10548823-24:head [write 766485~465633] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.792422146, 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075029242999999995, 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.005 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4062 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150173+0000", 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150435+0000", 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000262064 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150445+0000", 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8160000000000005e-06 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150450+0000", 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1880000000000001e-06 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150484+0000", 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150490+0000", 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6810000000000003e-06 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161686+0000", 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011196187 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161840+0000", 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015389000000000001 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.162103+0000", 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026333299999999998 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224927+0000", 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062823458999999998 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224993+0000", 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6667000000000004e-05 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225059+0000", 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6117000000000004e-05 2024-04-20T02:53:40.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225119+0000", 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9342000000000002e-05 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225204+0000", 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4844000000000003e-05 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.011 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4063 4.9 4:9c87d36d:::smithi10548823-24:head [write 1990637~647776] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.792139472000002, 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075097435000000004, 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4063 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:40.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150746+0000", 2024-04-20T02:53:40.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028951399999999999 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150753+0000", 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150757+0000", 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2300000000000002e-06 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150801+0000", 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3421000000000003e-05 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150811+0000", 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0581000000000001e-05 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177065+0000", 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02625367 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177164+0000", 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8900999999999994e-05 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177319+0000", 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000155437 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225048+0000", 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047728857 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225217+0000", 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168395 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225375+0000", 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157837 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225437+0000", 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2824999999999996e-05 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225555+0000", 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117169 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4064 4.9 4:9c87d36d:::smithi10548823-24:head [write 3299280~667205] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.791832718999999, 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112222372, 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.018 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4064 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150763+0000", 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151076+0000", 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000312355 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151083+0000", 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1010000000000003e-06 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151091+0000", 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0409999999999998e-06 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151127+0000", 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151133+0000", 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1099999999999999e-06 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200197+0000", 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049064000000000003 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200315+0000", 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117855 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200545+0000", 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022990700000000001 2024-04-20T02:53:40.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225595+0000", 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025050224999999999 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262670+0000", 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075460999999997 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262767+0000", 2024-04-20T02:53:40.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6479999999999998e-05 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262835+0000", 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999999e-05 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262986+0000", 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015129799999999999 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4065 4.9 4:9c87d36d:::smithi10548823-24:head [setxattr _header (57) in=7b,truncate 3966485] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.791492839, 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112193353, 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4065 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151103+0000", 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151107+0000", 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151111+0000", 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2309999999999998e-06 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151116+0000", 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4850000000000003e-06 2024-04-20T02:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151168+0000", 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2673000000000002e-05 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151179+0000", 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0112000000000001e-05 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225611+0000", 2024-04-20T02:53:40.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074432549000000001 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225689+0000", 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7612000000000002e-05 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225858+0000", 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016945400000000001 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262650+0000", 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036792126000000001 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262678+0000", 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8132000000000001e-05 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263077+0000", 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039868299999999999 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263141+0000", 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3448999999999995e-05 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263297+0000", 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015649699999999999 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4066 4.9 4:9c87d36d:::smithi10548823-24:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:40.031 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.791475982999998, 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112663448, 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4066 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151120+0000", 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151123+0000", 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4389999999999999e-06 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151128+0000", 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2449999999999998e-06 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151132+0000", 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9049999999999999e-06 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151198+0000", 2024-04-20T02:53:40.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5988000000000006e-05 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151205+0000", 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8770000000000004e-06 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226171+0000", 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074966844000000005 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226261+0000", 2024-04-20T02:53:40.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9975999999999994e-05 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263337+0000", 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075863000000001 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263443+0000", 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000105678 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263784+0000", 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034103200000000001 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4095 4.9 4:964390c0:::smithi10548823-36:head [write 450520~558231] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.791461912999999, 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112875926, 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.037 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4095 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151134+0000", 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151379+0000", 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024486000000000001 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151385+0000", 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1460000000000001e-06 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151389+0000", 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6610000000000002e-06 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151413+0000", 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3611000000000001e-05 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151419+0000", 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8780000000000003e-06 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226302+0000", 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074883540999999998 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226460+0000", 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015792000000000001 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226599+0000", 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013939599999999999 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262685+0000", 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036086115000000002 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262726+0000", 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0448999999999998e-05 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263833+0000", 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001106685 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263884+0000", 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1530999999999998e-05 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264011+0000", 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012641900000000001 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4096 4.9 4:964390c0:::smithi10548823-36:head [write 1483047~515125] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.043 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.791202378000001, 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11703358799999999, 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4096 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151644+0000", 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025029799999999998 2024-04-20T02:53:40.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151651+0000", 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.148e-06 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151655+0000", 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2769999999999999e-06 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151689+0000", 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711999999999998e-05 2024-04-20T02:53:40.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151698+0000", 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0359999999999995e-06 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245111+0000", 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093413164000000007 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245232+0000", 2024-04-20T02:53:40.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000120214 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245424+0000", 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019272500000000001 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263033+0000", 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017608461999999998 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266386+0000", 2024-04-20T02:53:40.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033530980000000001 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268218+0000", 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001832101 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268283+0000", 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5334999999999997e-05 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268428+0000", 2024-04-20T02:53:40.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014449999999999999 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4097 4.9 4:964390c0:::smithi10548823-36:head [setxattr _header (57) in=7b,truncate 2050520] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.790934425000003, 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.117031099, 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.050 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4097 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151665+0000", 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.614e-06 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151670+0000", 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.052 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151675+0000", 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0560000000000004e-06 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151738+0000", 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2447999999999996e-05 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151753+0000", 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5007e-05 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264044+0000", 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112291562 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264165+0000", 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121081 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264421+0000", 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025568599999999998 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266133+0000", 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001711668 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266755+0000", 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00062224100000000005 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268484+0000", 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001729169 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268545+0000", 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0365000000000002e-05 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268693+0000", 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014860000000000001 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4098 4.9 4:964390c0:::smithi10548823-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:40.056 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 47.790916942999999, 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11735205, 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4098 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151679+0000", 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151681+0000", 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3140000000000002e-06 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151686+0000", 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0209999999999999e-06 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151691+0000", 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4700000000000004e-06 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151779+0000", 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8399999999999994e-05 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151788+0000", 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7129999999999994e-06 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265294+0000", 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11350631899999999 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265391+0000", 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6861000000000007e-05 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268730+0000", 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003339102 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268841+0000", 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110708 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.269032+0000", 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019057899999999999 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.061 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.062 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.062 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.062 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.062 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:40.063 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:40.064 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:40.063+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:40.164 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:40.164+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:40.233 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:40.234 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:40.234 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:40.234 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:40.244 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:40.285 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:40.286+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:40.323 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:53:40.376 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:40.376+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:40.417 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:40.417 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:40.417 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:40.417 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:40.417 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:40.427 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:40.466 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:40.466+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:40.566 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:40.567+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:40.599 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:40.599 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:40.599 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:40.599 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:40.599 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.599 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:40.599 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:40.600 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 174.05414268999999, 2024-04-20T02:53:40.600 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:53:40.600 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.600 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.600 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.600 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.600 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.600 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:53:40.600 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.600 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.601 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.602 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.603 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.604 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.605 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:53:40.606 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 166.22559380199999, 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.607 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.608 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:53:40.609 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:53:40.610 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.611 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.612 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:53:40.613 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.614 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 166.20934162, 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.615 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.616 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:53:40.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:53:40.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.617 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:53:40.617 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:53:40.617 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.617 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.617 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:53:40.618 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.619 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 108.24964260100001, 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.620 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.621 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.622 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.623 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:53:40.624 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.625 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.626 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3872 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 779924~785847,stat] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.247239362000002, 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070878165000000007, 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3872 2024-04-20T02:53:40.627 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.628 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350641+0000", 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351179+0000", 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053797199999999995 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.629 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351196+0000", 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.66e-05 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351209+0000", 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2952000000000001e-05 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351306+0000", 2024-04-20T02:53:40.630 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7272e-05 2024-04-20T02:53:40.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351322+0000", 2024-04-20T02:53:40.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6416e-05 2024-04-20T02:53:40.631 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.631 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.631 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.631 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358164+0000", 2024-04-20T02:53:40.631 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068416019999999996 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358336+0000", 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017156499999999999 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358594+0000", 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025794499999999998 2024-04-20T02:53:40.632 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389148+0000", 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030554773 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409688+0000", 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020539497 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.633 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420893+0000", 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011205503 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420934+0000", 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1171000000000003e-05 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420998+0000", 2024-04-20T02:53:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3386999999999995e-05 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421521+0000", 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052313200000000005 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3873 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 2078257~580112] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:40.635 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.246656076000001, 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075737411000000004, 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3873 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.636 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351225+0000", 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:40.637 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351614+0000", 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038825099999999999 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351625+0000", 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1173e-05 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.638 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351634+0000", 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0489999999999993e-06 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351696+0000", 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1885999999999997e-05 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.639 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351708+0000", 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2702e-05 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389193+0000", 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037484574999999999 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389303+0000", 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110188 2024-04-20T02:53:40.640 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389563+0000", 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026034499999999999 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420846+0000", 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031283014999999997 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.641 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421592+0000", 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074529299999999995 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421632+0000", 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0677999999999999e-05 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.642 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426683+0000", 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050507119999999997 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426752+0000", 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8484000000000004e-05 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426964+0000", 2024-04-20T02:53:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021193500000000001 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3874 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 3301257~278667] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.246221075000001, 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075720365999999997, 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.644 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3874 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.645 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351660+0000", 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.646 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351852+0000", 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019116900000000001 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351861+0000", 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6050000000000004e-06 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351869+0000", 2024-04-20T02:53:40.647 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4780000000000002e-06 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351916+0000", 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7222999999999998e-05 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351927+0000", 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0791000000000001e-05 2024-04-20T02:53:40.648 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409721+0000", 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057793887000000002 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409840+0000", 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119837 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.649 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.410063+0000", 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022257200000000001 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421678+0000", 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011615448 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.650 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421721+0000", 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2191000000000003e-05 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.423989+0000", 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022682879999999998 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427051+0000", 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003061785 2024-04-20T02:53:40.651 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427157+0000", 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106684 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427381+0000", 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022416299999999999 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.652 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.653 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.653 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.653 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3875 4.10 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (57) in=7b,truncate 3579924] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:40.653 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:40.653 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.245999914000002, 2024-04-20T02:53:40.653 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075851731000000006, 2024-04-20T02:53:40.653 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.653 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3875 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.654 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.655 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351886+0000", 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.561e-06 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351893+0000", 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6100000000000002e-06 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351899+0000", 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8660000000000002e-06 2024-04-20T02:53:40.656 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352002+0000", 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103811 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352012+0000", 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4110000000000002e-06 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.657 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421749+0000", 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.069736690000000004 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421861+0000", 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112587 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422120+0000", 2024-04-20T02:53:40.658 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025878099999999997 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424547+0000", 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024270379999999999 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424960+0000", 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041259899999999998 2024-04-20T02:53:40.659 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426257+0000", 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001297231 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427489+0000", 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012320949999999999 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.660 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427562+0000", 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2825999999999997e-05 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427734+0000", 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017230699999999999 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.661 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3876 4.10 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.245978008000002, 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076230629999999994, 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3876 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.662 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.663 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351906+0000", 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6350000000000002e-06 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351912+0000", 2024-04-20T02:53:40.664 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8209999999999998e-06 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351918+0000", 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3839999999999996e-06 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352035+0000", 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117207 2024-04-20T02:53:40.665 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352043+0000", 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0169999999999996e-06 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422511+0000", 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070467815000000003 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.666 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422619+0000", 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010819600000000001 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427772+0000", 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005153398 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.667 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427900+0000", 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012740399999999999 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.428135+0000", 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235138 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.668 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4067 4.c 4:392c1e2f:::smithi10548823-44:head [write 783292~540882] snapc 47=[47,45,3f] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 48.455285799000002, 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072910900000000001, 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4067 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.669 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142595+0000", 2024-04-20T02:53:40.670 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142941+0000", 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034585799999999999 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142954+0000", 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3491e-05 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.671 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142965+0000", 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1139e-05 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143003+0000", 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7499000000000001e-05 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143011+0000", 2024-04-20T02:53:40.672 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5490000000000006e-06 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156033+0000", 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013021576999999999 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156242+0000", 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020885500000000001 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.673 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156474+0000", 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195900000000001 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177835+0000", 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021361594000000001 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.674 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178072+0000", 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000237304 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.214963+0000", 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036890158999999999 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215046+0000", 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3399000000000006e-05 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.675 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215507+0000", 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046127700000000001 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4068 4.c 4:392c1e2f:::smithi10548823-44:head [write 1932766~507787] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 48.454906856000001, 2024-04-20T02:53:40.676 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073156952999999997, 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4068 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.677 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.678 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143300+0000", 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000325225 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143308+0000", 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3860000000000007e-06 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143315+0000", 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7000000000000002e-06 2024-04-20T02:53:40.679 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143360+0000", 2024-04-20T02:53:40.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5475000000000001e-05 2024-04-20T02:53:40.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.680 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.680 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.680 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143373+0000", 2024-04-20T02:53:40.680 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2204e-05 2024-04-20T02:53:40.680 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178109+0000", 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034736547999999999 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178222+0000", 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112959 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.681 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178452+0000", 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000229686 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215864+0000", 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037412038000000002 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215878+0000", 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4029000000000001e-05 2024-04-20T02:53:40.682 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215929+0000", 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0738000000000001e-05 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215989+0000", 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0350999999999999e-05 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.683 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216132+0000", 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014332099999999999 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4069 4.c 4:392c1e2f:::smithi10548823-44:head [write 3210350~572013] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 48.454558042999999, 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108473219, 2024-04-20T02:53:40.684 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4069 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.685 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.686 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143664+0000", 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034090500000000002 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143673+0000", 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1119999999999996e-06 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143679+0000", 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3589999999999996e-06 2024-04-20T02:53:40.687 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143724+0000", 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4974e-05 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143734+0000", 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0275000000000001e-05 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.688 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197473+0000", 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053738971000000003 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197616+0000", 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014275900000000001 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197834+0000", 2024-04-20T02:53:40.689 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021806000000000001 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251517+0000", 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053683445000000003 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251579+0000", 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1255999999999998e-05 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.690 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251590+0000", 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1399e-05 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251657+0000", 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7039999999999995e-05 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251797+0000", 2024-04-20T02:53:40.691 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000140235 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4070 4.c 4:392c1e2f:::smithi10548823-44:head [setxattr _header (57) in=7b,truncate 3983292] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 48.454195255000002, 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108344988, 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.692 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4070 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:40.693 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143686+0000", 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143690+0000", 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7579999999999998e-06 2024-04-20T02:53:40.694 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143697+0000", 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0119999999999999e-06 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143702+0000", 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.78e-06 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.695 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143758+0000", 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640999999999998e-05 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143766+0000", 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9519999999999994e-06 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216172+0000", 2024-04-20T02:53:40.696 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072406561999999994 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216295+0000", 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122452 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216531+0000", 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023635999999999999 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.697 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251815+0000", 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035283760999999997 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251856+0000", 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1270999999999998e-05 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251892+0000", 2024-04-20T02:53:40.698 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251929+0000", 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7055999999999999e-05 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252032+0000", 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103183 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.699 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4071 4.c 4:392c1e2f:::smithi10548823-44:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 48.454160055999999, 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108670215, 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.700 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4071 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:40.701 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143721+0000", 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143727+0000", 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2510000000000002e-06 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.702 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143739+0000", 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2649e-05 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143749+0000", 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3340000000000008e-06 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.703 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143788+0000", 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9773e-05 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143796+0000", 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8720000000000002e-06 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216846+0000", 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073049823 2024-04-20T02:53:40.704 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216948+0000", 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101525 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252070+0000", 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035122911999999999 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.705 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252180+0000", 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010907499999999999 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252392+0000", 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021285400000000001 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.706 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4106 4.c 4:3d524298:::smithi10548823-18:head [write 484134~708729] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 48.454126950999999, 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108955254, 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4106 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.707 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:40.708 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144188+0000", 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000433771 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144198+0000", 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4970000000000007e-06 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.709 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144205+0000", 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3409999999999997e-06 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145714+0000", 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015088110000000001 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145727+0000", 2024-04-20T02:53:40.710 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2989e-05 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217011+0000", 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071283814000000001 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217168+0000", 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015707300000000001 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.711 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217397+0000", 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022881900000000001 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251870+0000", 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034473539999999997 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.712 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252459+0000", 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000589151 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252511+0000", 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1282999999999997e-05 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252569+0000", 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8294999999999997e-05 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.713 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252711+0000", 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000141435 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4107 4.c 4:3d524298:::smithi10548823-18:head [write 1883658~200476] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:40.714 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 48.453664885999999, 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.10880477099999999, 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4107 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.715 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144216+0000", 2024-04-20T02:53:40.716 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144346+0000", 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000129732 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144355+0000", 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4300000000000006e-06 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.717 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144361+0000", 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9150000000000001e-06 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145754+0000", 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0013929649999999999 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.718 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145763+0000", 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2990000000000002e-06 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243874+0000", 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.098111462999999996 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243990+0000", 2024-04-20T02:53:40.719 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0001156 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.244226+0000", 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235839 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252760+0000", 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085345439999999998 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.720 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252817+0000", 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6842000000000002e-05 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252829+0000", 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1817999999999999e-05 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252884+0000", 2024-04-20T02:53:40.721 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4610000000000001e-05 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253022+0000", 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013853800000000001 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.722 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4108 4.c 4:3d524298:::smithi10548823-18:head [setxattr _header (57) in=7b,truncate 2084134] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 48.453516462000003, 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.120324168, 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4108 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.723 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.724 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144365+0000", 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144369+0000", 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8500000000000004e-06 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144374+0000", 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0080000000000002e-06 2024-04-20T02:53:40.725 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144379+0000", 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8239999999999999e-06 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145790+0000", 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014115449999999999 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.726 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145800+0000", 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0429999999999996e-06 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253101+0000", 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.107301173 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253210+0000", 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109146 2024-04-20T02:53:40.727 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:40.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253456+0000", 2024-04-20T02:53:40.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024599700000000002 2024-04-20T02:53:40.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.728 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.728 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.728 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254623+0000", 2024-04-20T02:53:40.728 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001166718 2024-04-20T02:53:40.728 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.258220+0000", 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003597135 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264486+0000", 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062665819999999997 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.729 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264558+0000", 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1364000000000003e-05 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264690+0000", 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013227099999999999 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.730 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4109 4.c 4:3d524298:::smithi10548823-18:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 48.453498940000003, 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12051582399999999, 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4109 2024-04-20T02:53:40.731 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144385+0000", 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2869999999999999e-06 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144392+0000", 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5749999999999997e-06 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144397+0000", 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3589999999999999e-06 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145829+0000", 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00143191 2024-04-20T02:53:40.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145838+0000", 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254052+0000", 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108214029 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:40.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254156+0000", 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010343799999999999 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264726+0000", 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010570144 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264796+0000", 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9886000000000001e-05 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264899+0000", 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103336 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:4146 3.1 e153/152 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head v 153'228, mlcod=0'0)", 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 48.416102754000001, 2024-04-20T02:53:40.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066047175, 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181778+0000", 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181997+0000", 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021896 2024-04-20T02:53:40.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182011+0000", 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3339999999999999e-05 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182021+0000", 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.046e-05 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182068+0000", 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6931999999999997e-05 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182104+0000", 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6321e-05 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247786+0000", 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065681787000000005 2024-04-20T02:53:40.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247827+0000", 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0491000000000002e-05 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4183 3.2 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head [write 0~2392064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:40.742 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 46.946468582000001, 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064859071000000004, 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4183 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.743 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651396+0000", 2024-04-20T02:53:40.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999819 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652285+0000", 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088949500000000002 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652306+0000", 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0812e-05 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652333+0000", 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7086000000000001e-05 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652373+0000", 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9690000000000001e-05 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:40.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652499+0000", 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012642299999999999 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652608+0000", 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108661 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.702325+0000", 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049717048999999999 2024-04-20T02:53:40.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.711665+0000", 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093403370000000006 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716188+0000", 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004522997 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:40.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716227+0000", 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9240999999999999e-05 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716273+0000", 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5228000000000001e-05 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:40.749 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:40.750 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:40.751 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:40.751+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:40.752 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:40.852 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:40.853+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:40.894 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:40.894 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:40.894 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:40.895 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:40.904 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:40.953 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:40.954+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:40.966 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:53:41.031 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:41.032 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:41.032 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:41.032 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:41.032 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:41.041 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:41.086 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:41.087+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:41.111 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:41.129 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:41.149 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:41.155 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:41.156+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:41.168 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:41.176 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:41.176 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:41.176 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:41.176 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:41.176 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.176 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:41.176 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:41.176 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.44014160899999, 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.177 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:53:41.178 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:53:41.179 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.180 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:53:41.181 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.408852548, 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:53:41.182 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.183 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.184 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.185 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.186 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.187 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.40416209200001, 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:41.188 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.189 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.190 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.191 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:41.192 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.193 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.397405623, 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:41.194 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.195 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.196 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.197 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.198 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:53:41.198 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:53:41.198 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.198 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:41.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:53:41.199 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:53:41.199 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.199 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.199 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.199 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.200 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:41.201 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.387590112, 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.202 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:53:41.203 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:53:41.204 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:53:41.205 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.206 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.38654618000001, 2024-04-20T02:53:41.207 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.208 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.209 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:53:41.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:53:41.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.210 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.210 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.210 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:53:41.210 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:53:41.210 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.211 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.211 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.211 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:53:41.211 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:53:41.211 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.212 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.212 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:53:41.212 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:53:41.212 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.212 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.213 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:41.213 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:53:41.213 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:53:41.214 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.214 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.214 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:41.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:53:41.215 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:53:41.215 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.215 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.215 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.215 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:53:41.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:53:41.216 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.216 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.216 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:41.216 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:53:41.216 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:53:41.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.217 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.217 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:53:41.217 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:53:41.217 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.217 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.218 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.218 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:53:41.218 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:53:41.218 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.38652501999999, 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:53:41.219 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:41.220 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:53:41.221 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.222 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.223 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.17895443899999, 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.224 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.225 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.226 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.227 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.228 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.228 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.228 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.228 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:41.228 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:41.228 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.16475979399999, 2024-04-20T02:53:41.228 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:53:41.228 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.228 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.228 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.229 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.230 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:53:41.231 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.232 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.15879253700001, 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:41.233 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:53:41.234 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.235 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.236 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.15038653400001, 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.237 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:53:41.238 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:53:41.239 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:53:41.240 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 179.04675449300001, 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.241 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.242 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.243 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.244 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 178.816158183, 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.245 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.246 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:53:41.247 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:53:41.248 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 178.789214034, 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:53:41.249 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.250 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.251 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.252 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:41.253 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 178.763855767, 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.254 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:53:41.255 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:53:41.256 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.257 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 178.73571532299999, 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:41.258 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:53:41.259 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:53:41.260 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.261 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 178.69834671800001, 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.262 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.263 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.264 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.265 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 178.698049795, 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.266 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.267 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.268 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.269 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 178.69296667699999, 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:41.270 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:53:41.271 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:53:41.272 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:53:41.273 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 178.66596411800001, 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:41.274 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:41.275 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.276 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:41.277 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:41.278 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:41.278 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:53:41.278 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:53:41.278 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.278 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.278 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.278 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:41.278 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:41.278 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:41.280 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:41.280 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:41.280 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:41.280 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:41.280 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:41.281 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:41.281 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:41.281 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:41.282 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:41.282 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:41.282 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:41.282 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:41.282 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:41.282 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:41.280+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:41.380 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:41.380+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:41.450 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:41.450 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:41.460 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:41.501 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:41.502+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:41.582 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:41.582+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:41.628 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:41.629 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:41.639 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:41.683 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:41.684+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:41.694 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 3, 6, 7, 5, 2, 4] 2024-04-20T02:53:41.694 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:53:41.694 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-20T02:53:41.694 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.609286 2024-04-20T02:53:41.695 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 0.6092859635693947 2024-04-20T02:53:41.784 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:41.785+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:41.806 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:41.806 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:41.816 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:41.885 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:41.886+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:41.944 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:41.944 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:41.955 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:41.985 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:41.986+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:42.082 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:42.083 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:42.086 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:42.086+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:42.093 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:42.187 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:42.188+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:42.260 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:42.261 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:42.271 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:42.288 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:42.288+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:42.332 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.5 primary-affinity to 0.609286 (8399302) 2024-04-20T02:53:42.336 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:53:42.389 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:42.389+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:42.414 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:42.414 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:42.426 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:42.489 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:42.491+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:42.574 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:42.574 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:42.585 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:42.590 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:42.591+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:42.691 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:42.692+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:42.755 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:42.756 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:42.766 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:42.809 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:42.810+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:42.893 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:42.894+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:42.896 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:42.896 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:42.906 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:42.994 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:42.994+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:43.074 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:43.074 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:43.085 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:43.095 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:43.096+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:43.196 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:43.197+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:43.254 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:43.254 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:43.264 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:43.283 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:43.308 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:43.309+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:43.336 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:53:43.398 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:43.399+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:43.404 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:43.404 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:43.416 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:43.460 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:43.480 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:43.499 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:43.499+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:43.500 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:43.520 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:43.539 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:43.559 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:43.579 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:43.587 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:43.587 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:43.588 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:43.588 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:43.598 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:43.599 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:43.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:43.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:43.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:43.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:43.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:43.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:43.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:43.600 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:43.601 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:43.601+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:43.700 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:43.701+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:43.731 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:43.731 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:43.731 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:43.731 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:43.731 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.731 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:43.731 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:43.731 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 172.79725215600001, 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.732 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.733 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.734 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.736 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 172.79699195000001, 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:53:43.737 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.738 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.739 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.740 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.741 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.742 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 172.79697048200001, 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:53:43.743 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.744 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.745 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.746 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.747 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3771 4.b 4:d4308fef:::smithi10548823-6:head [write 2548663~763022] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.547941824999995, 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072733780999999997, 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.748 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3771 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.749 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181891+0000", 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182301+0000", 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040985099999999998 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.750 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182317+0000", 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6305999999999999e-05 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182329+0000", 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1337e-05 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198720+0000", 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016391606 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.751 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198807+0000", 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7319999999999997e-05 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198925+0000", 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117345 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218062+0000", 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019137134 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.752 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234990+0000", 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016928367 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254488+0000", 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019497460000000001 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254571+0000", 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3029999999999996e-05 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.753 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254627+0000", 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5865000000000001e-05 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3772 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3395047] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.547495252999994, 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072419708999999999, 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.754 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3772 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.755 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182343+0000", 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0099999999999997e-06 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.756 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182349+0000", 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9569999999999999e-06 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182354+0000", 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.826e-06 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217228+0000", 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034874555000000002 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.757 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217312+0000", 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3478000000000003e-05 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217431+0000", 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119512 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234969+0000", 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017537833999999999 2024-04-20T02:53:43.758 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235012+0000", 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3411000000000002e-05 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254638+0000", 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019625815000000001 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254682+0000", 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3945e-05 2024-04-20T02:53:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254759+0000", 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6882e-05 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3773 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:43.760 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 64.547476646000007, 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072538680999999994, 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3773 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:43.761 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.762 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182360+0000", 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3309999999999998e-06 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182366+0000", 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.981e-06 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.763 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182371+0000", 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5390000000000001e-06 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217932+0000", 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035561700000000002 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218020+0000", 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7352e-05 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.764 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254778+0000", 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036757971 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254819+0000", 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1525e-05 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254897+0000", 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7621999999999996e-05 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.765 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4057 4.b 4:d4308fef:::smithi10548823-6:head [write 749678~790384,stat] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.586621905999998, 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074563737000000005, 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.766 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4057 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.767 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143210+0000", 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143760+0000", 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055015999999999995 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.768 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143772+0000", 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1345000000000001e-05 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143781+0000", 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0929999999999992e-06 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143895+0000", 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113998 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.769 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143906+0000", 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1195999999999999e-05 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.153590+0000", 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096836000000000005 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155611+0000", 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002021012 2024-04-20T02:53:43.770 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217691+0000", 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062080244 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217777+0000", 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5583999999999999e-05 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.771 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4058 4.b 4:d4308fef:::smithi10548823-6:head [write 2332155~504118] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.586040879999999, 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074090323999999999, 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4058 2024-04-20T02:53:43.772 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.773 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143793+0000", 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144104+0000", 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000310971 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144112+0000", 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1990000000000006e-06 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.774 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144118+0000", 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.443e-06 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144162+0000", 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3066999999999999e-05 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144172+0000", 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0083e-05 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.775 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155911+0000", 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011739657000000001 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155993+0000", 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1754000000000004e-05 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217810+0000", 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061816958999999998 2024-04-20T02:53:43.776 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217884+0000", 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4098999999999996e-05 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4059 4.b 4:d4308fef:::smithi10548823-6:head [write 3271857~549469] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.585704718999999, 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073860173000000001, 2024-04-20T02:53:43.777 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4059 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.778 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144129+0000", 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144439+0000", 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030924000000000001 2024-04-20T02:53:43.779 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144447+0000", 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8930000000000005e-06 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144455+0000", 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1410000000000005e-06 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144505+0000", 2024-04-20T02:53:43.780 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0698000000000001e-05 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144520+0000", 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4605e-05 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156034+0000", 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011513809 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156130+0000", 2024-04-20T02:53:43.781 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6039999999999995e-05 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217915+0000", 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061785501999999999 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217990+0000", 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4770000000000004e-05 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.782 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4060 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3949678] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.58536934, 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085188106999999999, 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.783 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4060 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.784 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:43.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.785 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.785 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.785 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144469+0000", 2024-04-20T02:53:43.785 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.946e-06 2024-04-20T02:53:43.785 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144476+0000", 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7909999999999999e-06 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144482+0000", 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.601e-06 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144803+0000", 2024-04-20T02:53:43.786 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032103699999999998 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144813+0000", 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0183999999999999e-05 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156168+0000", 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011354523 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156254+0000", 2024-04-20T02:53:43.787 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5706999999999998e-05 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218023+0000", 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061769566999999997 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218137+0000", 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113912 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218377+0000", 2024-04-20T02:53:43.788 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023941000000000001 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225107+0000", 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067304510000000001 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225121+0000", 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4341e-05 2024-04-20T02:53:43.789 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229481+0000", 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043599880000000004 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229522+0000", 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0224e-05 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.790 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229653+0000", 2024-04-20T02:53:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013190500000000001 2024-04-20T02:53:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4061 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:43.791 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.585229722999998, 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085431572999999997, 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4061 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.792 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144604+0000", 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144608+0000", 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1080000000000003e-06 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.793 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144618+0000", 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5149999999999995e-06 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144625+0000", 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8260000000000004e-06 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144846+0000", 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022160699999999999 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.794 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144854+0000", 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0399999999999993e-06 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156327+0000", 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011472178 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156457+0000", 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013012599999999999 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.795 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218704+0000", 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062247080000000003 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218795+0000", 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1391000000000006e-05 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229703+0000", 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010907834999999999 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.796 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229818+0000", 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115423 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.230037+0000", 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021803600000000001 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4146 3.1 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head [write 0~1129472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:43.797 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.549633745999998, 2024-04-20T02:53:43.798 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067860845000000003, 2024-04-20T02:53:43.798 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.798 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.798 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.798 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.798 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.798 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4146 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:43.799 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180199+0000", 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180439+0000", 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000239375 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.800 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180449+0000", 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0220999999999999e-05 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180456+0000", 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0090000000000001e-06 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181170+0000", 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071358900000000004 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.801 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181331+0000", 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016178500000000001 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,5", 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181490+0000", 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158703 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225260+0000", 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043770166999999999 2024-04-20T02:53:43.802 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226398+0000", 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001138189 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247971+0000", 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021572728999999999 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248021+0000", 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0176999999999997e-05 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.803 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248062+0000", 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0472000000000001e-05 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4623 4.3 4:c07c6daa:::smithi10548823-7:head [write 2772312~34523] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.579705898, 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074418926999999996, 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.804 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4623 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:43.805 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150128+0000", 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150143+0000", 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5166e-05 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.806 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150147+0000", 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.924e-06 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150151+0000", 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.777e-06 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150180+0000", 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8472000000000001e-05 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.807 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150184+0000", 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4349999999999999e-06 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187063+0000", 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036878474000000001 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187145+0000", 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2107999999999994e-05 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.808 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187280+0000", 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135384 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189223+0000", 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019430820000000001 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189411+0000", 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000188061 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.809 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224429+0000", 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035017386999999997 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224487+0000", 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8884000000000001e-05 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224548+0000", 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0238000000000002e-05 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.810 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4624 4.3 4:c07c6daa:::smithi10548823-7:head [setxattr _header (56) in=7b,truncate 2806835] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.579679652999999, 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074609524999999996, 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4624 2024-04-20T02:53:43.811 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.812 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150158+0000", 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9799999999999998e-06 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150161+0000", 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8140000000000001e-06 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.813 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150164+0000", 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5329999999999999e-06 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150197+0000", 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2926999999999998e-05 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150201+0000", 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2989999999999998e-06 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.814 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188464+0000", 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038262276999999997 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188576+0000", 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112413 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188756+0000", 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000180206 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.815 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189459+0000", 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070307500000000005 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224557+0000", 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035097721999999998 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224633+0000", 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5798999999999997e-05 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.816 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224690+0000", 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6784999999999999e-05 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224765+0000", 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4969999999999995e-05 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4625 4.3 4:c07c6daa:::smithi10548823-7:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:43.817 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.579667691000001, 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074770387999999993, 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4625 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.819 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150169+0000", 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7680000000000001e-06 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150171+0000", 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8779999999999998e-06 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150175+0000", 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2909999999999999e-06 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.820 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150215+0000", 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0074000000000003e-05 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150219+0000", 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.177e-06 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189474+0000", 2024-04-20T02:53:43.821 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039255274999999999 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189531+0000", 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7188999999999997e-05 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224802+0000", 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035270981999999999 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.822 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224861+0000", 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8276999999999998e-05 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224937+0000", 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6689999999999999e-05 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4661 4.3 4:cd374330:::smithi10548823-1:head [write 490870~618799] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.579656932999999, 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074896037999999998, 2024-04-20T02:53:43.823 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4661 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.824 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150452+0000", 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027429199999999998 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.825 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150458+0000", 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5810000000000002e-06 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150463+0000", 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7029999999999997e-06 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150486+0000", 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3308000000000001e-05 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.826 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150491+0000", 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6179999999999997e-06 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189565+0000", 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039073926000000002 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189713+0000", 2024-04-20T02:53:43.827 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014765700000000001 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189930+0000", 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021749700000000001 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224591+0000", 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034661210999999997 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.828 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224775+0000", 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000184091 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224987+0000", 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021128100000000001 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225040+0000", 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3395e-05 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.829 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225074+0000", 2024-04-20T02:53:43.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3699999999999999e-05 2024-04-20T02:53:43.830 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.830 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.830 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.830 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.830 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.830 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4662 4.3 4:cd374330:::smithi10548823-1:head [write 1592676~774144] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:43.830 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:43.830 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.579366049000001, 2024-04-20T02:53:43.830 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075836991000000006, 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4662 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.831 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150468+0000", 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150808+0000", 2024-04-20T02:53:43.832 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033972899999999999 2024-04-20T02:53:43.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150815+0000", 2024-04-20T02:53:43.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8159999999999996e-06 2024-04-20T02:53:43.833 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.833 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.833 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.833 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150819+0000", 2024-04-20T02:53:43.833 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150855+0000", 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150886+0000", 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0876000000000001e-05 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.204837+0000", 2024-04-20T02:53:43.834 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053950873000000003 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.204916+0000", 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9008e-05 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.205039+0000", 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012294000000000001 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224780+0000", 2024-04-20T02:53:43.835 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019741771000000002 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225703+0000", 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00092290600000000003 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226196+0000", 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049284199999999995 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.836 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226250+0000", 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4413000000000002e-05 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226306+0000", 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.516e-05 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4663 4.3 4:cd374330:::smithi10548823-1:head [write 2793074~97796] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.579010476000001, 2024-04-20T02:53:43.837 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075759200999999998, 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4663 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.838 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150883+0000", 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9308999999999997e-05 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.839 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150888+0000", 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1139999999999997e-06 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150892+0000", 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5899999999999999e-06 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150924+0000", 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2289000000000002e-05 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.840 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150929+0000", 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0259999999999998e-06 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221580+0000", 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070650828999999998 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221708+0000", 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012845700000000001 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.841 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221897+0000", 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000188789 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225650+0000", 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037528980000000002 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226379+0000", 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072901200000000002 2024-04-20T02:53:43.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226492+0000", 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112905 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226526+0000", 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711000000000003e-05 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.843 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226583+0000", 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7624999999999998e-05 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4664 4.3 4:cd374330:::smithi10548823-1:head [setxattr _header (56) in=7b,truncate 2890870] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.578936846000001, 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076283431999999998, 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:43.844 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4664 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:43.845 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150900+0000", 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.571e-06 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150903+0000", 2024-04-20T02:53:43.846 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9409999999999999e-06 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150907+0000", 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5599999999999998e-06 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150988+0000", 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1544000000000004e-05 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151006+0000", 2024-04-20T02:53:43.847 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7754999999999999e-05 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225114+0000", 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074107889999999996 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225189+0000", 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4851999999999994e-05 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225305+0000", 2024-04-20T02:53:43.848 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011592899999999999 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225800+0000", 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049498200000000004 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226624+0000", 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00082437000000000005 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227055+0000", 2024-04-20T02:53:43.849 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043119499999999999 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227113+0000", 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7500999999999997e-05 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227181+0000", 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8625000000000001e-05 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4665 4.3 4:cd374330:::smithi10548823-1:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 22.578912450000001, 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076413855000000003, 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4665 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.851 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.852 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150924+0000", 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8190000000000001e-06 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150929+0000", 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6400000000000002e-06 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150933+0000", 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5039999999999998e-06 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.853 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151025+0000", 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2354000000000004e-05 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151040+0000", 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708999999999999e-05 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225553+0000", 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074513356000000003 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.854 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225640+0000", 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6335000000000006e-05 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227206+0000", 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001566448 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227261+0000", 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4907000000000001e-05 2024-04-20T02:53:43.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:43.856 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:43.856 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:43.856 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227336+0000", 2024-04-20T02:53:43.856 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5050999999999997e-05 2024-04-20T02:53:43.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.856 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.856 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:43.856 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:43.856 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:43.858 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:43.858 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:43.858+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:43.959 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:43.960+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:44.027 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:44.027 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:44.027 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:44.027 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:44.038 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:44.060 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:44.060+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:44.161 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:44.162+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:44.205 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:44.206 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:44.206 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:44.206 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:44.206 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:44.216 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:44.262 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:44.262+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 177.76265218, 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.349 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:44.350 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.351 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.352 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.353 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.354 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 177.74192286900001, 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.355 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.356 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.356 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.356 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.357 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.358 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.359 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:53:44.360 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:53:44.361 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 177.73267111800001, 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.362 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.363 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.364 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.365 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.366 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.367 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 169.935671902, 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.368 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.369 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:53:44.370 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:53:44.371 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 169.92966711400001, 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:44.372 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:53:44.373 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:53:44.374 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:53:44.375 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4029 4.13 4:cf11ccc0:::smithi10548823-4:head [write 699257~564637,stat] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.792124598000001, 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.86467550400000004, 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.376 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4029 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.377 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555666+0000", 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555809+0000", 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014306900000000001 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555832+0000", 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3680999999999999e-05 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.378 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555846+0000", 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3654e-05 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555913+0000", 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6736999999999995e-05 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556058+0000", 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014538100000000001 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.379 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556392+0000", 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033393099999999998 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578661+0000", 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022268553999999999 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578702+0000", 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1533999999999999e-05 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.380 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578747+0000", 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4858999999999999e-05 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578811+0000", 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3520000000000002e-05 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605743+0000", 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026932418999999999 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.381 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420312+0000", 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81456849600000003 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420343+0000", 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1911e-05 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4030 4.13 4:cf11ccc0:::smithi10548823-4:head [write 1873062~640438] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:44.382 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.785211476999997, 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85780235299999996, 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4030 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.383 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562579+0000", 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562714+0000", 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013475 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.384 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562728+0000", 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3516000000000001e-05 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562736+0000", 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5690000000000005e-06 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578835+0000", 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016099156 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.385 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578957+0000", 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121094 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.579123+0000", 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016647100000000001 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605727+0000", 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026603643999999999 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.386 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605749+0000", 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2070999999999999e-05 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605799+0000", 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0417000000000002e-05 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605841+0000", 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2344e-05 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.387 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605907+0000", 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5135999999999994e-05 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420373+0000", 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81446675899999998 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420383+0000", 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0108999999999999e-05 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.388 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4031 4.13 4:cf11ccc0:::smithi10548823-4:head [write 3305941~593316] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.777649902, 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85027709799999995, 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4031 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.389 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570142+0000", 2024-04-20T02:53:44.390 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570277+0000", 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135606 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570288+0000", 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0719e-05 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570294+0000", 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2519999999999996e-06 2024-04-20T02:53:44.391 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592786+0000", 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022491220999999999 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592911+0000", 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012546599999999999 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.593118+0000", 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000207011 2024-04-20T02:53:44.392 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605878+0000", 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012760298999999999 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605934+0000", 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5813999999999998e-05 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605963+0000", 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9275e-05 2024-04-20T02:53:44.393 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606612+0000", 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00064813999999999996 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613782+0000", 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071705670000000001 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420411+0000", 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.80662907399999995 2024-04-20T02:53:44.394 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420420+0000", 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5790000000000004e-06 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4032 4.13 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (56) in=7b,truncate 3899257] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.777493474000003, 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85015831100000006, 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.395 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4032 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:44.396 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570302+0000", 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1700000000000001e-06 2024-04-20T02:53:44.397 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570305+0000", 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.976e-06 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570308+0000", 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8899999999999999e-06 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605985+0000", 2024-04-20T02:53:44.398 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035677121999999999 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606066+0000", 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0797000000000006e-05 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606211+0000", 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014569999999999999 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607013+0000", 2024-04-20T02:53:44.399 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00080160599999999995 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607046+0000", 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2938999999999998e-05 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607086+0000", 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0203000000000001e-05 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607157+0000", 2024-04-20T02:53:44.400 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0621e-05 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613833+0000", 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066762269999999999 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420448+0000", 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.806615478 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420457+0000", 2024-04-20T02:53:44.401 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9779999999999994e-06 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4033 4.13 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.777481600000002, 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85021192700000003, 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.402 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4033 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.403 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570312+0000", 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419999999999999e-06 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570316+0000", 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3730000000000001e-06 2024-04-20T02:53:44.404 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570318+0000", 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6680000000000001e-06 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606541+0000", 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036222797000000001 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606598+0000", 2024-04-20T02:53:44.405 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6629999999999998e-05 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420515+0000", 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81391740099999998 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420523+0000", 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6799999999999993e-06 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4034 4.13 4:cfd1e4d4:::smithi10548823-9:head [rollback 3f] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.777471962, 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85017147900000001, 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4034 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.407 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570320+0000", 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.408 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570322+0000", 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.37e-06 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570325+0000", 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7800000000000001e-06 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570328+0000", 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5379999999999998e-06 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.409 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606632+0000", 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036304075999999998 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606716+0000", 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3689000000000005e-05 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420484+0000", 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81376812700000001 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.410 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420492+0000", 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1179999999999992e-06 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4062 4.9 4:9c87d36d:::smithi10548823-24:head [write 766485~465633] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.197618181000003, 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075029242999999995, 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.411 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4062 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:44.412 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150173+0000", 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150435+0000", 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000262064 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150445+0000", 2024-04-20T02:53:44.413 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8160000000000005e-06 2024-04-20T02:53:44.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150450+0000", 2024-04-20T02:53:44.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1880000000000001e-06 2024-04-20T02:53:44.414 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.414 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.414 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.414 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150484+0000", 2024-04-20T02:53:44.414 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150490+0000", 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6810000000000003e-06 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161686+0000", 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011196187 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.415 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161840+0000", 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015389000000000001 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.162103+0000", 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026333299999999998 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224927+0000", 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062823458999999998 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.416 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224993+0000", 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6667000000000004e-05 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225059+0000", 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6117000000000004e-05 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225119+0000", 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9342000000000002e-05 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.417 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225204+0000", 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4844000000000003e-05 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4063 4.9 4:9c87d36d:::smithi10548823-24:head [write 1990637~647776] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.197335506999998, 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075097435000000004, 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.418 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4063 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.419 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150746+0000", 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028951399999999999 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150753+0000", 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.420 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150757+0000", 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2300000000000002e-06 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150801+0000", 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3421000000000003e-05 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150811+0000", 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0581000000000001e-05 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.421 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177065+0000", 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02625367 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177164+0000", 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8900999999999994e-05 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177319+0000", 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000155437 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.422 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225048+0000", 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047728857 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225217+0000", 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168395 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225375+0000", 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157837 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.423 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225437+0000", 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2824999999999996e-05 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225555+0000", 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117169 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4064 4.9 4:9c87d36d:::smithi10548823-24:head [write 3299280~667205] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:44.424 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.197028754000002, 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112222372, 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4064 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.425 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150763+0000", 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151076+0000", 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000312355 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.426 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151083+0000", 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1010000000000003e-06 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151091+0000", 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0409999999999998e-06 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151127+0000", 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.427 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151133+0000", 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1099999999999999e-06 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200197+0000", 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049064000000000003 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200315+0000", 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117855 2024-04-20T02:53:44.428 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200545+0000", 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022990700000000001 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225595+0000", 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025050224999999999 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262670+0000", 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075460999999997 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.429 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262767+0000", 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6479999999999998e-05 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262835+0000", 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999999e-05 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262986+0000", 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015129799999999999 2024-04-20T02:53:44.430 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4065 4.9 4:9c87d36d:::smithi10548823-24:head [setxattr _header (57) in=7b,truncate 3966485] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.196688874000003, 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112193353, 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4065 2024-04-20T02:53:44.431 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151103+0000", 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:44.432 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151107+0000", 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151111+0000", 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2309999999999998e-06 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151116+0000", 2024-04-20T02:53:44.433 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4850000000000003e-06 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151168+0000", 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2673000000000002e-05 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151179+0000", 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0112000000000001e-05 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225611+0000", 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074432549000000001 2024-04-20T02:53:44.434 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225689+0000", 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7612000000000002e-05 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225858+0000", 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016945400000000001 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262650+0000", 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036792126000000001 2024-04-20T02:53:44.435 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262678+0000", 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8132000000000001e-05 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263077+0000", 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039868299999999999 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263141+0000", 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3448999999999995e-05 2024-04-20T02:53:44.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263297+0000", 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015649699999999999 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4066 4.9 4:9c87d36d:::smithi10548823-24:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.196672018000001, 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112663448, 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:44.437 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4066 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151120+0000", 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151123+0000", 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4389999999999999e-06 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151128+0000", 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2449999999999998e-06 2024-04-20T02:53:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151132+0000", 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9049999999999999e-06 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151198+0000", 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5988000000000006e-05 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151205+0000", 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8770000000000004e-06 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226171+0000", 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074966844000000005 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226261+0000", 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9975999999999994e-05 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263337+0000", 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075863000000001 2024-04-20T02:53:44.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263443+0000", 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000105678 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263784+0000", 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034103200000000001 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4095 4.9 4:964390c0:::smithi10548823-36:head [write 450520~558231] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.196657948000002, 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112875926, 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4095 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.443 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151134+0000", 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151379+0000", 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024486000000000001 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151385+0000", 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1460000000000001e-06 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151389+0000", 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6610000000000002e-06 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151413+0000", 2024-04-20T02:53:44.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3611000000000001e-05 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151419+0000", 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8780000000000003e-06 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226302+0000", 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074883540999999998 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226460+0000", 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015792000000000001 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226599+0000", 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013939599999999999 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262685+0000", 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036086115000000002 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262726+0000", 2024-04-20T02:53:44.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0448999999999998e-05 2024-04-20T02:53:44.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263833+0000", 2024-04-20T02:53:44.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001106685 2024-04-20T02:53:44.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263884+0000", 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1530999999999998e-05 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264011+0000", 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012641900000000001 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.450 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4096 4.9 4:964390c0:::smithi10548823-36:head [write 1483047~515125] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.196398412999997, 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11703358799999999, 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4096 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151644+0000", 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025029799999999998 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151651+0000", 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.148e-06 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151655+0000", 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2769999999999999e-06 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151689+0000", 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711999999999998e-05 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151698+0000", 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0359999999999995e-06 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245111+0000", 2024-04-20T02:53:44.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093413164000000007 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245232+0000", 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000120214 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245424+0000", 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019272500000000001 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.455 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263033+0000", 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017608461999999998 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266386+0000", 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033530980000000001 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268218+0000", 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001832101 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268283+0000", 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5334999999999997e-05 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268428+0000", 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014449999999999999 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4097 4.9 4:964390c0:::smithi10548823-36:head [setxattr _header (57) in=7b,truncate 2050520] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.196130459999999, 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.117031099, 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4097 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151665+0000", 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.614e-06 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151670+0000", 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151675+0000", 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0560000000000004e-06 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151738+0000", 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2447999999999996e-05 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151753+0000", 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5007e-05 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264044+0000", 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112291562 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264165+0000", 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121081 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264421+0000", 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025568599999999998 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266133+0000", 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001711668 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.462 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266755+0000", 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00062224100000000005 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268484+0000", 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001729169 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268545+0000", 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0365000000000002e-05 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268693+0000", 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014860000000000001 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4098 4.9 4:964390c0:::smithi10548823-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.196112978000002, 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11735205, 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:44.464 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4098 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151679+0000", 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151681+0000", 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3140000000000002e-06 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151686+0000", 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0209999999999999e-06 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151691+0000", 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4700000000000004e-06 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151779+0000", 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8399999999999994e-05 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151788+0000", 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7129999999999994e-06 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265294+0000", 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11350631899999999 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265391+0000", 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6861000000000007e-05 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268730+0000", 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003339102 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268841+0000", 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110708 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.269032+0000", 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019057899999999999 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:44.469 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:44.471 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:44.471 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:44.472+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:44.571 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:44.573+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:44.641 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:44.641 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:44.641 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:44.641 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:44.651 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:44.673 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:44.673+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:44.773 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:44.774+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:44.820 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:44.820 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:44.820 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:44.820 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:44.820 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:44.830 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:44.914 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:44.915+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:44.975 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:44.976+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 178.45614744299999, 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.001 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:45.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:53:45.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:53:45.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 170.62759855499999, 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:53:45.007 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:45.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:53:45.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:53:45.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:53:45.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.012 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.012 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:53:45.012 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:53:45.012 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.012 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.013 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 170.611346373, 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.014 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.015 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.016 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.017 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 112.651647354, 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.018 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:45.019 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:53:45.020 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:53:45.021 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:53:45.022 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:53:45.023 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3872 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 779924~785847,stat] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.649244115000002, 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070878165000000007, 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.024 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3872 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.025 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350641+0000", 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351179+0000", 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053797199999999995 2024-04-20T02:53:45.026 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351196+0000", 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.66e-05 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351209+0000", 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2952000000000001e-05 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351306+0000", 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7272e-05 2024-04-20T02:53:45.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351322+0000", 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6416e-05 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358164+0000", 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068416019999999996 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358336+0000", 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017156499999999999 2024-04-20T02:53:45.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358594+0000", 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025794499999999998 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389148+0000", 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030554773 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409688+0000", 2024-04-20T02:53:45.029 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020539497 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420893+0000", 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011205503 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420934+0000", 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1171000000000003e-05 2024-04-20T02:53:45.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420998+0000", 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3386999999999995e-05 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421521+0000", 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052313200000000005 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.031 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3873 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 2078257~580112] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.648660829000001, 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075737411000000004, 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3873 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.032 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351225+0000", 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.033 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351614+0000", 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038825099999999999 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351625+0000", 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1173e-05 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351634+0000", 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0489999999999993e-06 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.034 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351696+0000", 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1885999999999997e-05 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351708+0000", 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2702e-05 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389193+0000", 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037484574999999999 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.035 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389303+0000", 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110188 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389563+0000", 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026034499999999999 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420846+0000", 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031283014999999997 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421592+0000", 2024-04-20T02:53:45.036 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074529299999999995 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421632+0000", 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0677999999999999e-05 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426683+0000", 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050507119999999997 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426752+0000", 2024-04-20T02:53:45.037 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8484000000000004e-05 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426964+0000", 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021193500000000001 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3874 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 3301257~278667] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.648225828000001, 2024-04-20T02:53:45.038 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075720365999999997, 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3874 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.039 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351660+0000", 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351852+0000", 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019116900000000001 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.040 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351861+0000", 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6050000000000004e-06 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351869+0000", 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4780000000000002e-06 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351916+0000", 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7222999999999998e-05 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.041 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351927+0000", 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0791000000000001e-05 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409721+0000", 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057793887000000002 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409840+0000", 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119837 2024-04-20T02:53:45.042 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.410063+0000", 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022257200000000001 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421678+0000", 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011615448 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421721+0000", 2024-04-20T02:53:45.043 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2191000000000003e-05 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.423989+0000", 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022682879999999998 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427051+0000", 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003061785 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427157+0000", 2024-04-20T02:53:45.044 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106684 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427381+0000", 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022416299999999999 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3875 4.10 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (57) in=7b,truncate 3579924] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.648004667000002, 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075851731000000006, 2024-04-20T02:53:45.045 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3875 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.046 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351886+0000", 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.561e-06 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.047 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351893+0000", 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6100000000000002e-06 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351899+0000", 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8660000000000002e-06 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352002+0000", 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103811 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.048 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352012+0000", 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4110000000000002e-06 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421749+0000", 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.069736690000000004 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421861+0000", 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112587 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:45.049 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422120+0000", 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025878099999999997 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424547+0000", 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024270379999999999 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424960+0000", 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041259899999999998 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.050 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426257+0000", 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001297231 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427489+0000", 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012320949999999999 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427562+0000", 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2825999999999997e-05 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.051 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427734+0000", 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017230699999999999 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3876 4.10 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.647982761000002, 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076230629999999994, 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.052 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3876 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.053 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351906+0000", 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6350000000000002e-06 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351912+0000", 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8209999999999998e-06 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.054 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351918+0000", 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3839999999999996e-06 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352035+0000", 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117207 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352043+0000", 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0169999999999996e-06 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.055 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422511+0000", 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070467815000000003 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422619+0000", 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010819600000000001 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427772+0000", 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005153398 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.056 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427900+0000", 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012740399999999999 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.428135+0000", 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235138 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4067 4.c 4:392c1e2f:::smithi10548823-44:head [write 783292~540882] snapc 47=[47,45,3f] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.857290552000002, 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072910900000000001, 2024-04-20T02:53:45.057 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4067 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.058 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142595+0000", 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142941+0000", 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034585799999999999 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.059 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142954+0000", 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3491e-05 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142965+0000", 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1139e-05 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143003+0000", 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7499000000000001e-05 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143011+0000", 2024-04-20T02:53:45.060 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5490000000000006e-06 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156033+0000", 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013021576999999999 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156242+0000", 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020885500000000001 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156474+0000", 2024-04-20T02:53:45.061 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195900000000001 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177835+0000", 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021361594000000001 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178072+0000", 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000237304 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.214963+0000", 2024-04-20T02:53:45.062 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036890158999999999 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215046+0000", 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3399000000000006e-05 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215507+0000", 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046127700000000001 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.063 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4068 4.c 4:392c1e2f:::smithi10548823-44:head [write 1932766~507787] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.856911609000001, 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073156952999999997, 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4068 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:45.064 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143300+0000", 2024-04-20T02:53:45.065 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000325225 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143308+0000", 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3860000000000007e-06 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143315+0000", 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7000000000000002e-06 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143360+0000", 2024-04-20T02:53:45.066 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5475000000000001e-05 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143373+0000", 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2204e-05 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178109+0000", 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034736547999999999 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178222+0000", 2024-04-20T02:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112959 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178452+0000", 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000229686 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215864+0000", 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037412038000000002 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215878+0000", 2024-04-20T02:53:45.068 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4029000000000001e-05 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215929+0000", 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0738000000000001e-05 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215989+0000", 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0350999999999999e-05 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216132+0000", 2024-04-20T02:53:45.069 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014332099999999999 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4069 4.c 4:392c1e2f:::smithi10548823-44:head [write 3210350~572013] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.856562795999999, 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108473219, 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.070 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4069 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.071 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143664+0000", 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034090500000000002 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143673+0000", 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1119999999999996e-06 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.072 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143679+0000", 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3589999999999996e-06 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143724+0000", 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4974e-05 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143734+0000", 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0275000000000001e-05 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.073 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197473+0000", 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053738971000000003 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197616+0000", 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014275900000000001 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197834+0000", 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021806000000000001 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.074 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251517+0000", 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053683445000000003 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251579+0000", 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1255999999999998e-05 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251590+0000", 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1399e-05 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.075 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251657+0000", 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7039999999999995e-05 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251797+0000", 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000140235 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4070 4.c 4:392c1e2f:::smithi10548823-44:head [setxattr _header (57) in=7b,truncate 3983292] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.856200008000002, 2024-04-20T02:53:45.076 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108344988, 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4070 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.077 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143686+0000", 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143690+0000", 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7579999999999998e-06 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.078 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143697+0000", 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0119999999999999e-06 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143702+0000", 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.78e-06 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143758+0000", 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640999999999998e-05 2024-04-20T02:53:45.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143766+0000", 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9519999999999994e-06 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216172+0000", 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072406561999999994 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.080 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216295+0000", 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122452 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216531+0000", 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023635999999999999 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251815+0000", 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035283760999999997 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251856+0000", 2024-04-20T02:53:45.081 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1270999999999998e-05 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251892+0000", 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251929+0000", 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7055999999999999e-05 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.082 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252032+0000", 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103183 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4071 4.c 4:392c1e2f:::smithi10548823-44:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.856164808999999, 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108670215, 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.083 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4071 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.084 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143721+0000", 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143727+0000", 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2510000000000002e-06 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.085 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143739+0000", 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2649e-05 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143749+0000", 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3340000000000008e-06 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143788+0000", 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9773e-05 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.086 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143796+0000", 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8720000000000002e-06 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216846+0000", 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073049823 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216948+0000", 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101525 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252070+0000", 2024-04-20T02:53:45.087 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035122911999999999 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252180+0000", 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010907499999999999 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252392+0000", 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021285400000000001 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.088 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4106 4.c 4:3d524298:::smithi10548823-18:head [write 484134~708729] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.856131703999999, 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108955254, 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4106 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144188+0000", 2024-04-20T02:53:45.090 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000433771 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144198+0000", 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4970000000000007e-06 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144205+0000", 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3409999999999997e-06 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145714+0000", 2024-04-20T02:53:45.091 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015088110000000001 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145727+0000", 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2989e-05 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217011+0000", 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071283814000000001 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.092 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217168+0000", 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015707300000000001 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217397+0000", 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022881900000000001 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251870+0000", 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034473539999999997 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252459+0000", 2024-04-20T02:53:45.093 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000589151 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252511+0000", 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1282999999999997e-05 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252569+0000", 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8294999999999997e-05 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.094 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252711+0000", 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000141435 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4107 4.c 4:3d524298:::smithi10548823-18:head [write 1883658~200476] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.855669638999998, 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.10880477099999999, 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.095 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4107 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144216+0000", 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144346+0000", 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000129732 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144355+0000", 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4300000000000006e-06 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.097 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144361+0000", 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9150000000000001e-06 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145754+0000", 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0013929649999999999 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145763+0000", 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2990000000000002e-06 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.098 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243874+0000", 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.098111462999999996 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243990+0000", 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0001156 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.244226+0000", 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235839 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252760+0000", 2024-04-20T02:53:45.099 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085345439999999998 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252817+0000", 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6842000000000002e-05 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252829+0000", 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1817999999999999e-05 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252884+0000", 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4610000000000001e-05 2024-04-20T02:53:45.100 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253022+0000", 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013853800000000001 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4108 4.c 4:3d524298:::smithi10548823-18:head [setxattr _header (57) in=7b,truncate 2084134] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.855521215000003, 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.120324168, 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.101 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4108 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.102 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144365+0000", 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144369+0000", 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8500000000000004e-06 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144374+0000", 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0080000000000002e-06 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.103 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144379+0000", 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8239999999999999e-06 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145790+0000", 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014115449999999999 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145800+0000", 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0429999999999996e-06 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.104 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253101+0000", 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.107301173 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253210+0000", 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109146 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253456+0000", 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024599700000000002 2024-04-20T02:53:45.105 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254623+0000", 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001166718 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.258220+0000", 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003597135 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264486+0000", 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062665819999999997 2024-04-20T02:53:45.106 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264558+0000", 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1364000000000003e-05 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264690+0000", 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013227099999999999 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.107 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4109 4.c 4:3d524298:::smithi10548823-18:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.855503693000003, 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12051582399999999, 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4109 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.108 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.109 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144385+0000", 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2869999999999999e-06 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144392+0000", 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5749999999999997e-06 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144397+0000", 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3589999999999999e-06 2024-04-20T02:53:45.110 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145829+0000", 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00143191 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145838+0000", 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254052+0000", 2024-04-20T02:53:45.111 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108214029 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254156+0000", 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010343799999999999 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264726+0000", 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010570144 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264796+0000", 2024-04-20T02:53:45.112 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9886000000000001e-05 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264899+0000", 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103336 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:4146 3.1 e153/152 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head v 153'228, mlcod=0'0)", 2024-04-20T02:53:45.113 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 52.818107507000001, 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066047175, 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:45.114 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181778+0000", 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181997+0000", 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021896 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182011+0000", 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3339999999999999e-05 2024-04-20T02:53:45.115 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182021+0000", 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.046e-05 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182068+0000", 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6931999999999997e-05 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182104+0000", 2024-04-20T02:53:45.116 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6321e-05 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247786+0000", 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065681787000000005 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247827+0000", 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0491000000000002e-05 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.117 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4183 3.2 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head [write 0~2392064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 51.348473335000001, 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064859071000000004, 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4183 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.118 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651396+0000", 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999819 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.119 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652285+0000", 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088949500000000002 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652306+0000", 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0812e-05 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652333+0000", 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7086000000000001e-05 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.120 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652373+0000", 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9690000000000001e-05 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652499+0000", 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012642299999999999 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652608+0000", 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108661 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:45.121 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.702325+0000", 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049717048999999999 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.711665+0000", 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093403370000000006 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716188+0000", 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004522997 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:45.122 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716227+0000", 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9240999999999999e-05 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716273+0000", 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5228000000000001e-05 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:45.123 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:45.125 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:45.125 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:45.126+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:45.226 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:45.226+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:45.266 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:45.266 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:45.266 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:45.266 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:45.277 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:45.327 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:45.328+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:45.349 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-20T02:53:45.427 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:45.428+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:45.448 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:45.448 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:45.448 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:45.448 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:45.448 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:45.459 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:45.542 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:45.543+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:45.613 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-20T02:53:45.629 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:45.629 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:45.629 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:45.629 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:45.629 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.893372599, 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.630 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.631 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:53:45.632 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:53:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:53:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.633 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:53:45.634 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.635 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:53:45.636 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.86208353800001, 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:45.637 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.638 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.639 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.640 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:53:45.641 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:53:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.85739308199999, 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:53:45.644 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.645 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:45.645 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.645 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.645 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.645 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.645 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:53:45.646 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.647 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.648 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.649 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:53:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:53:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:53:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:53:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.650 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.85063661300001, 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.651 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:45.652 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.653 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:53:45.654 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.655 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:53:45.656 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.657 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.84082110200001, 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:53:45.658 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.659 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.660 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:53:45.661 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.662 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.663 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.664 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.83977716999999, 2024-04-20T02:53:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:53:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.665 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.666 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:53:45.667 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.668 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:53:45.669 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:53:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:53:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:53:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:53:45.670 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.671 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.83975601, 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.672 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.673 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:53:45.674 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.675 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:53:45.676 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.677 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.632185429, 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.678 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:53:45.679 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.680 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:53:45.681 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.682 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.617990784, 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:45.683 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.684 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:53:45.685 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.686 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.687 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.61202352699999, 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.688 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:53:45.689 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.690 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.691 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.60361752399999, 2024-04-20T02:53:45.692 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:45.693 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.694 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.695 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.696 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.49998548299999, 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:53:45.697 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:45.698 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.699 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:53:45.700 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.701 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.26938917300001, 2024-04-20T02:53:45.702 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.703 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.704 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:53:45.705 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.706 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.24244502400001, 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:53:45.707 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.708 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.709 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:53:45.710 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.711 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.217086757, 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.712 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.713 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.714 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:53:45.715 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.716 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.188946313, 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.717 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.718 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:53:45.719 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.720 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:53:45.721 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.15157770799999, 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.722 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.723 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:53:45.724 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.725 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:53:45.726 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.15128078500001, 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.727 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:53:45.728 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.729 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:53:45.730 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.731 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.146197667, 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.732 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.733 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.734 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.735 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.736 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 183.11919510800001, 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:45.737 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.738 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:53:45.739 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.740 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:45.741 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:45.743 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:45.744 INFO:tasks.ceph.osd.4.smithi138.stderr:2024-04-20T02:53:45.743+0000 7f9d37071700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 66265) UID: 0 2024-04-20T02:53:45.758 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:45.777 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:45.795 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:45.814 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:45.833 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:45.843 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:45.844+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:45.852 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:45.870 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:45.889 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:45.891 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:45.891 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:45.891 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:45.891 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:45.891 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:45.891 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:45.891 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:45.891 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:45.914 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:45.914 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:45.924 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:45.944 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:45.945+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:46.045 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:46.045+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:46.092 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:46.092 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:46.102 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:46.145 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:46.146+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:46.230 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:46.230 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:46.241 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:46.284 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:46.285+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:46.347 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:46.348+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:46.355 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is set 2024-04-20T02:53:46.369 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:46.369 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:46.380 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:46.449 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:46.449+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:46.548 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:46.548 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:46.550 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:46.550+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:46.559 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:46.650 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:46.651+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:46.686 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:46.686 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:46.696 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:46.751 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:46.751+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:46.830 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:46.830 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:46.840 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:46.881 INFO:tasks.ceph.osd.2.smithi105.stderr:2024-04-20T02:53:46.881+0000 7f038eb2f700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 66771) UID: 0 2024-04-20T02:53:46.954 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:46.954+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:47.009 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:47.009 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:47.020 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:47.054 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:47.055+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:47.155 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:47.156+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:47.188 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:47.188 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:47.198 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:47.256 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:47.257+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:47.332 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:47.332 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:47.343 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:47.350 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 5] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 3, 6, 7, 5, 2, 4] 2024-04-20T02:53:47.350 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-20T02:53:47.350 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-20T02:53:47.350 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:53:47.384 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:47.385+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:47.457 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:47.458+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:47.512 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:47.512 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:47.522 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:47.558 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:47.558+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:47.659 INFO:tasks.ceph.osd.6.smithi138.stderr:2024-04-20T02:53:47.659+0000 7f78d9fe1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 43323) UID: 0 2024-04-20T02:53:47.682 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:53:47.683 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:53:47.694 INFO:teuthology.orchestra.run.smithi138.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:47.694 INFO:teuthology.orchestra.run.smithi138.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:47.695 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-04-20T02:53:47.483980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":219655603,"num_objects":178,"num_object_clones":7,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":4128,"num_read_kb":3828471,"num_write":2451,"num_write_kb":622342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":284251718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":17,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":647988,"kb_used_data":435996,"kb_used_omap":9,"kb_used_meta":211894,"kb_avail":565583052,"statfs":{"total":579820584960,"available":579157045248,"internally_reserved":0,"allocated":446459904,"data_stored":465937713,"data_compressed":11470125,"data_compressed_allocated":12410880,"data_compressed_original":27955200,"omap_allocated":9619,"internal_metadata":216980077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.151560"},"pg_stats":[{"pgid":"4.1","version":"184'139","reported_seq":711,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148523+0000","last_change":"2024-04-20T02:53:29.153095+0000","last_active":"2024-04-20T02:53:41.148523+0000","last_peered":"2024-04-20T02:53:41.148523+0000","last_clean":"2024-04-20T02:53:41.148523+0000","last_became_active":"2024-04-20T02:53:29.152688+0000","last_became_peered":"2024-04-20T02:53:29.152688+0000","last_unstale":"2024-04-20T02:53:41.148523+0000","last_undegraded":"2024-04-20T02:53:41.148523+0000","last_fullsized":"2024-04-20T02:53:41.148523+0000","mapping_epoch":154,"log_start":"184'139","ondisk_log_start":"184'139","created":22,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'131","last_scrub_stamp":"2024-04-20T02:52:54.612743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.612743+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":368,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:59.591499+0000","scrub_duration":0.0022862500000000001,"objects_trimmed":1,"snaptrim_duration":0.011020622000000001,"stat_sum":{"num_bytes":8084332,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":252,"num_read_kb":318196,"num_write":63,"num_write_kb":15810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":28106829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":155,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148036+0000","last_change":"2024-04-20T02:53:30.912852+0000","last_active":"2024-04-20T02:53:41.148036+0000","last_peered":"2024-04-20T02:53:41.148036+0000","last_clean":"2024-04-20T02:53:41.148036+0000","last_became_active":"2024-04-20T02:53:18.152172+0000","last_became_peered":"2024-04-20T02:53:18.152172+0000","last_unstale":"2024-04-20T02:53:41.148036+0000","last_undegraded":"2024-04-20T02:53:41.148036+0000","last_fullsized":"2024-04-20T02:53:41.148036+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:32.689737+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"186'205","reported_seq":816,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.147526+0000","last_change":"2024-04-20T02:53:33.343239+0000","last_active":"2024-04-20T02:53:41.147526+0000","last_peered":"2024-04-20T02:53:41.147526+0000","last_clean":"2024-04-20T02:53:41.147526+0000","last_became_active":"2024-04-20T02:53:33.159428+0000","last_became_peered":"2024-04-20T02:53:33.159428+0000","last_unstale":"2024-04-20T02:53:41.147526+0000","last_undegraded":"2024-04-20T02:53:41.147526+0000","last_fullsized":"2024-04-20T02:53:41.147526+0000","mapping_epoch":176,"log_start":"186'205","ondisk_log_start":"186'205","created":22,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'205","last_scrub_stamp":"2024-04-20T02:53:33.342996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:33.342996+0000","objects_scrubbed":4,"log_size":0,"log_dups_size":343,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:49.442548+0000","scrub_duration":0.0033447149999999998,"objects_trimmed":0,"snaptrim_duration":0.042836615000000001,"stat_sum":{"num_bytes":9545632,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":378,"num_read_kb":480412,"num_write":113,"num_write_kb":33005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33476197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":224,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.952087+0000","last_change":"2024-04-20T02:53:19.107136+0000","last_active":"2024-04-20T02:53:41.952087+0000","last_peered":"2024-04-20T02:53:41.952087+0000","last_clean":"2024-04-20T02:53:41.952087+0000","last_became_active":"2024-04-20T02:52:51.153987+0000","last_became_peered":"2024-04-20T02:52:51.153987+0000","last_unstale":"2024-04-20T02:53:41.952087+0000","last_undegraded":"2024-04-20T02:53:41.952087+0000","last_fullsized":"2024-04-20T02:53:41.952087+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:24.234001+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":224,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688545+0000","last_change":"2024-04-20T02:52:53.937048+0000","last_active":"2024-04-20T02:53:42.688545+0000","last_peered":"2024-04-20T02:53:42.688545+0000","last_clean":"2024-04-20T02:53:42.688545+0000","last_became_active":"2024-04-20T02:52:51.154174+0000","last_became_peered":"2024-04-20T02:52:51.154174+0000","last_unstale":"2024-04-20T02:53:42.688545+0000","last_undegraded":"2024-04-20T02:53:42.688545+0000","last_fullsized":"2024-04-20T02:53:42.688545+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:53.936957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:53.936957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.656218+0000","scrub_duration":0.0013534090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"187'315","reported_seq":886,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688554+0000","last_change":"2024-04-20T02:53:23.159920+0000","last_active":"2024-04-20T02:53:42.688554+0000","last_peered":"2024-04-20T02:53:42.688554+0000","last_clean":"2024-04-20T02:53:42.688554+0000","last_became_active":"2024-04-20T02:53:21.153338+0000","last_became_peered":"2024-04-20T02:53:21.153338+0000","last_unstale":"2024-04-20T02:53:42.688554+0000","last_undegraded":"2024-04-20T02:53:42.688554+0000","last_fullsized":"2024-04-20T02:53:42.688554+0000","mapping_epoch":47,"log_start":"186'312","ondisk_log_start":"186'312","created":22,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'243","last_scrub_stamp":"2024-04-20T02:52:54.597931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.597931+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":557,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:59.861555+0000","scrub_duration":0.0028458709999999998,"objects_trimmed":2,"snaptrim_duration":0.0095485269999999994,"stat_sum":{"num_bytes":15467220,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":413,"num_read_kb":563626,"num_write":89,"num_write_kb":25274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":19243742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":620,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688396+0000","last_change":"2024-04-20T02:53:08.766997+0000","last_active":"2024-04-20T02:53:42.688396+0000","last_peered":"2024-04-20T02:53:42.688396+0000","last_clean":"2024-04-20T02:53:42.688396+0000","last_became_active":"2024-04-20T02:52:52.148432+0000","last_became_peered":"2024-04-20T02:52:52.148432+0000","last_unstale":"2024-04-20T02:53:42.688396+0000","last_undegraded":"2024-04-20T02:53:42.688396+0000","last_fullsized":"2024-04-20T02:53:42.688396+0000","mapping_epoch":152,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:29.132762+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":99,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688379+0000","last_change":"2024-04-20T02:52:32.918930+0000","last_active":"2024-04-20T02:53:42.688379+0000","last_peered":"2024-04-20T02:53:42.688379+0000","last_clean":"2024-04-20T02:53:42.688379+0000","last_became_active":"2024-04-20T02:52:22.261878+0000","last_became_peered":"2024-04-20T02:52:22.261878+0000","last_unstale":"2024-04-20T02:53:42.688379+0000","last_undegraded":"2024-04-20T02:53:42.688379+0000","last_fullsized":"2024-04-20T02:53:42.688379+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:32.918813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.918813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014288059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"186'153","reported_seq":742,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.951966+0000","last_change":"2024-04-20T02:53:31.855215+0000","last_active":"2024-04-20T02:53:41.951966+0000","last_peered":"2024-04-20T02:53:41.951966+0000","last_clean":"2024-04-20T02:53:41.951966+0000","last_became_active":"2024-04-20T02:53:16.151498+0000","last_became_peered":"2024-04-20T02:53:16.151498+0000","last_unstale":"2024-04-20T02:53:41.951966+0000","last_undegraded":"2024-04-20T02:53:41.951966+0000","last_fullsized":"2024-04-20T02:53:41.951966+0000","mapping_epoch":176,"log_start":"185'149","ondisk_log_start":"185'149","created":22,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'153","last_scrub_stamp":"2024-04-20T02:53:31.855077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:31.855077+0000","objects_scrubbed":5,"log_size":4,"log_dups_size":236,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.406736+0000","scrub_duration":0.0052908479999999999,"objects_trimmed":0,"snaptrim_duration":0.041643649999999997,"stat_sum":{"num_bytes":9717018,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":174,"num_read_kb":188998,"num_write":46,"num_write_kb":11974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22843153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":616,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.147464+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:53:41.147464+0000","last_peered":"2024-04-20T02:53:41.147464+0000","last_clean":"2024-04-20T02:53:41.147464+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:53:41.147464+0000","last_undegraded":"2024-04-20T02:53:41.147464+0000","last_fullsized":"2024-04-20T02:53:41.147464+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":367,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688477+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:53:42.688477+0000","last_peered":"2024-04-20T02:53:42.688477+0000","last_clean":"2024-04-20T02:53:42.688477+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:53:42.688477+0000","last_undegraded":"2024-04-20T02:53:42.688477+0000","last_fullsized":"2024-04-20T02:53:42.688477+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":200,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148216+0000","last_change":"2024-04-20T02:53:08.364116+0000","last_active":"2024-04-20T02:53:41.148216+0000","last_peered":"2024-04-20T02:53:41.148216+0000","last_clean":"2024-04-20T02:53:41.148216+0000","last_became_active":"2024-04-20T02:52:22.265487+0000","last_became_peered":"2024-04-20T02:52:22.265487+0000","last_unstale":"2024-04-20T02:53:41.148216+0000","last_undegraded":"2024-04-20T02:53:41.148216+0000","last_fullsized":"2024-04-20T02:53:41.148216+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.792324+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"186'105","reported_seq":372,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148275+0000","last_change":"2024-04-20T02:53:25.163140+0000","last_active":"2024-04-20T02:53:41.148275+0000","last_peered":"2024-04-20T02:53:41.148275+0000","last_clean":"2024-04-20T02:53:41.148275+0000","last_became_active":"2024-04-20T02:53:18.152319+0000","last_became_peered":"2024-04-20T02:53:18.152319+0000","last_unstale":"2024-04-20T02:53:41.148275+0000","last_undegraded":"2024-04-20T02:53:41.148275+0000","last_fullsized":"2024-04-20T02:53:41.148275+0000","mapping_epoch":178,"log_start":"184'101","ondisk_log_start":"184'101","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'86","last_scrub_stamp":"2024-04-20T02:52:32.901998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.901998+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":131,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:43.974508+0000","scrub_duration":0.0024332220000000001,"objects_trimmed":0,"snaptrim_duration":0.99220328899999999,"stat_sum":{"num_bytes":5395179,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":35504,"num_write":16,"num_write_kb":3789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1543958,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":545,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.951810+0000","last_change":"2024-04-20T02:52:42.201016+0000","last_active":"2024-04-20T02:53:41.951810+0000","last_peered":"2024-04-20T02:53:41.951810+0000","last_clean":"2024-04-20T02:53:41.951810+0000","last_became_active":"2024-04-20T02:52:25.272853+0000","last_became_peered":"2024-04-20T02:52:25.272853+0000","last_unstale":"2024-04-20T02:53:41.951810+0000","last_undegraded":"2024-04-20T02:53:41.951810+0000","last_fullsized":"2024-04-20T02:53:41.951810+0000","mapping_epoch":126,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'188","last_scrub_stamp":"2024-04-20T02:52:42.200865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:52:42.200865+0000","objects_scrubbed":31,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.320740+0000","scrub_duration":0.0052191850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":15696807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":102,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.147533+0000","last_change":"2024-04-20T02:52:56.212555+0000","last_active":"2024-04-20T02:53:41.147533+0000","last_peered":"2024-04-20T02:53:41.147533+0000","last_clean":"2024-04-20T02:53:41.147533+0000","last_became_active":"2024-04-20T02:52:22.272822+0000","last_became_peered":"2024-04-20T02:52:22.272822+0000","last_unstale":"2024-04-20T02:53:41.147533+0000","last_undegraded":"2024-04-20T02:53:41.147533+0000","last_fullsized":"2024-04-20T02:53:41.147533+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:56.212430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.212430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:20.820102+0000","scrub_duration":0.0013799870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"184'130","reported_seq":728,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148502+0000","last_change":"2024-04-20T02:53:23.161348+0000","last_active":"2024-04-20T02:53:41.148502+0000","last_peered":"2024-04-20T02:53:41.148502+0000","last_clean":"2024-04-20T02:53:41.148502+0000","last_became_active":"2024-04-20T02:52:46.141983+0000","last_became_peered":"2024-04-20T02:52:46.141983+0000","last_unstale":"2024-04-20T02:53:41.148502+0000","last_undegraded":"2024-04-20T02:53:41.148502+0000","last_fullsized":"2024-04-20T02:53:41.148502+0000","mapping_epoch":147,"log_start":"179'124","ondisk_log_start":"179'124","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'119","last_scrub_stamp":"2024-04-20T02:52:56.564632+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.564632+0000","objects_scrubbed":4,"log_size":6,"log_dups_size":310,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:22.552034+0000","scrub_duration":0.0028929759999999998,"objects_trimmed":1,"snaptrim_duration":0.010818365,"stat_sum":{"num_bytes":2840449,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":43301,"num_write":4,"num_write_kb":1274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":561,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688486+0000","last_change":"2024-04-20T02:53:18.737801+0000","last_active":"2024-04-20T02:53:42.688486+0000","last_peered":"2024-04-20T02:53:42.688486+0000","last_clean":"2024-04-20T02:53:42.688486+0000","last_became_active":"2024-04-20T02:53:11.150822+0000","last_became_peered":"2024-04-20T02:53:11.150822+0000","last_unstale":"2024-04-20T02:53:42.688486+0000","last_undegraded":"2024-04-20T02:53:42.688486+0000","last_fullsized":"2024-04-20T02:53:42.688486+0000","mapping_epoch":171,"log_start":"175'232","ondisk_log_start":"175'232","created":21,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":238,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.609708+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":18551655,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":200,"state":"peering","last_fresh":"2024-04-20T02:53:42.332078+0000","last_change":"2024-04-20T02:50:39.261258+0000","last_active":"2024-04-20T02:50:26.619512+0000","last_peered":"2024-04-20T02:50:26.619512+0000","last_clean":"2024-04-20T02:50:26.619512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-20T02:53:42.332078+0000","last_undegraded":"2024-04-20T02:53:42.332078+0000","last_fullsized":"2024-04-20T02:53:42.332078+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:00.725766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"185'48","reported_seq":529,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.951925+0000","last_change":"2024-04-20T02:53:33.880323+0000","last_active":"2024-04-20T02:53:41.951925+0000","last_peered":"2024-04-20T02:53:41.951925+0000","last_clean":"2024-04-20T02:53:41.951925+0000","last_became_active":"2024-04-20T02:53:17.156757+0000","last_became_peered":"2024-04-20T02:53:17.156757+0000","last_unstale":"2024-04-20T02:53:41.951925+0000","last_undegraded":"2024-04-20T02:53:41.951925+0000","last_fullsized":"2024-04-20T02:53:41.951925+0000","mapping_epoch":141,"log_start":"169'45","ondisk_log_start":"169'45","created":22,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'48","last_scrub_stamp":"2024-04-20T02:53:33.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:33.880214+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:59.742146+0000","scrub_duration":0.0018263730000000001,"objects_trimmed":0,"snaptrim_duration":0.039989539999999997,"stat_sum":{"num_bytes":2991712,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":274,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148814+0000","last_change":"2024-04-20T02:53:25.404436+0000","last_active":"2024-04-20T02:53:41.148814+0000","last_peered":"2024-04-20T02:53:41.148814+0000","last_clean":"2024-04-20T02:53:41.148814+0000","last_became_active":"2024-04-20T02:53:25.402753+0000","last_became_peered":"2024-04-20T02:53:25.402753+0000","last_unstale":"2024-04-20T02:53:41.148814+0000","last_undegraded":"2024-04-20T02:53:41.148814+0000","last_fullsized":"2024-04-20T02:53:41.148814+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_clean_scrub_stamp":"2024-04-20T02:52:43.686760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.573670+0000","scrub_duration":0.0018670150000000001,"objects_trimmed":0,"snaptrim_duration":0.001125371,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":204,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.952537+0000","last_change":"2024-04-20T02:52:18.363613+0000","last_active":"2024-04-20T02:53:41.952537+0000","last_peered":"2024-04-20T02:53:41.952537+0000","last_clean":"2024-04-20T02:53:41.952537+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:53:41.952537+0000","last_undegraded":"2024-04-20T02:53:41.952537+0000","last_fullsized":"2024-04-20T02:53:41.952537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:18.363433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:18.363433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016555929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"186'174","reported_seq":699,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148201+0000","last_change":"2024-04-20T02:53:33.417307+0000","last_active":"2024-04-20T02:53:41.148201+0000","last_peered":"2024-04-20T02:53:41.148201+0000","last_clean":"2024-04-20T02:53:41.148201+0000","last_became_active":"2024-04-20T02:52:13.145577+0000","last_became_peered":"2024-04-20T02:52:13.145577+0000","last_unstale":"2024-04-20T02:53:41.148201+0000","last_undegraded":"2024-04-20T02:53:41.148201+0000","last_fullsized":"2024-04-20T02:53:41.148201+0000","mapping_epoch":62,"log_start":"186'170","ondisk_log_start":"186'170","created":22,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'174","last_scrub_stamp":"2024-04-20T02:53:33.417175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:33.417175+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":328,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:39.243430+0000","scrub_duration":0.0025256810000000001,"objects_trimmed":0,"snaptrim_duration":0.00066820099999999999,"stat_sum":{"num_bytes":10729969,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":90,"num_read_kb":103553,"num_write":46,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"195'513","reported_seq":1223,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.951772+0000","last_change":"2024-04-20T02:53:37.203422+0000","last_active":"2024-04-20T02:53:41.951772+0000","last_peered":"2024-04-20T02:53:41.951772+0000","last_clean":"2024-04-20T02:53:41.951772+0000","last_became_active":"2024-04-20T02:53:11.151224+0000","last_became_peered":"2024-04-20T02:53:11.151224+0000","last_unstale":"2024-04-20T02:53:41.951772+0000","last_undegraded":"2024-04-20T02:53:41.951772+0000","last_fullsized":"2024-04-20T02:53:41.951772+0000","mapping_epoch":171,"log_start":"194'509","ondisk_log_start":"194'509","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'294","last_scrub_stamp":"2024-04-20T02:52:16.813424+0000","last_deep_scrub":"98'280","last_deep_scrub_stamp":"2024-04-20T02:51:56.947305+0000","last_clean_scrub_stamp":"2024-04-20T02:52:16.813424+0000","objects_scrubbed":7,"log_size":4,"log_dups_size":1130,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034018329999999999,"objects_trimmed":0,"snaptrim_duration":1.0562096670000001,"stat_sum":{"num_bytes":18824302,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":223,"num_read_kb":242421,"num_write":112,"num_write_kb":32076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":34386920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"184'120","reported_seq":605,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148561+0000","last_change":"2024-04-20T02:53:23.158044+0000","last_active":"2024-04-20T02:53:41.148561+0000","last_peered":"2024-04-20T02:53:41.148561+0000","last_clean":"2024-04-20T02:53:41.148561+0000","last_became_active":"2024-04-20T02:53:18.153426+0000","last_became_peered":"2024-04-20T02:53:18.153426+0000","last_unstale":"2024-04-20T02:53:41.148561+0000","last_undegraded":"2024-04-20T02:53:41.148561+0000","last_fullsized":"2024-04-20T02:53:41.148561+0000","mapping_epoch":178,"log_start":"177'116","ondisk_log_start":"177'116","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'53","last_scrub_stamp":"2024-04-20T02:52:05.810627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:05.810627+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":116,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022334680000000002,"objects_trimmed":0,"snaptrim_duration":0.00042807000000000001,"stat_sum":{"num_bytes":6343443,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":204,"num_read_kb":230945,"num_write":64,"num_write_kb":16950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"184'350","reported_seq":933,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688396+0000","last_change":"2024-04-20T02:53:30.856293+0000","last_active":"2024-04-20T02:53:42.688396+0000","last_peered":"2024-04-20T02:53:42.688396+0000","last_clean":"2024-04-20T02:53:42.688396+0000","last_became_active":"2024-04-20T02:53:11.147630+0000","last_became_peered":"2024-04-20T02:53:11.147630+0000","last_unstale":"2024-04-20T02:53:42.688396+0000","last_undegraded":"2024-04-20T02:53:42.688396+0000","last_fullsized":"2024-04-20T02:53:42.688396+0000","mapping_epoch":171,"log_start":"179'344","ondisk_log_start":"179'344","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'350","last_scrub_stamp":"2024-04-20T02:53:30.856060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.856060+0000","objects_scrubbed":5,"log_size":6,"log_dups_size":344,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.470953+0000","scrub_duration":0.0031057490000000001,"objects_trimmed":1,"snaptrim_duration":0.0080676189999999998,"stat_sum":{"num_bytes":11923149,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":388,"num_read_kb":461817,"num_write":178,"num_write_kb":48621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":23150965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"178'159","reported_seq":667,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.210611+0000","last_change":"2024-04-20T02:53:42.210611+0000","last_active":"2024-04-20T02:53:42.210611+0000","last_peered":"2024-04-20T02:53:42.210611+0000","last_clean":"2024-04-20T02:53:42.210611+0000","last_became_active":"2024-04-20T02:53:11.148476+0000","last_became_peered":"2024-04-20T02:53:11.148476+0000","last_unstale":"2024-04-20T02:53:42.210611+0000","last_undegraded":"2024-04-20T02:53:42.210611+0000","last_fullsized":"2024-04-20T02:53:42.210611+0000","mapping_epoch":171,"log_start":"173'155","ondisk_log_start":"173'155","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'159","last_scrub_stamp":"2024-04-20T02:53:42.210503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.210503+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":155,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:58.448617+0000","scrub_duration":0.0024829349999999999,"objects_trimmed":0,"snaptrim_duration":0.010578568,"stat_sum":{"num_bytes":4773320,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":178,"num_read_kb":193067,"num_write":90,"num_write_kb":24779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":14051707,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"188'288","reported_seq":943,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.994920+0000","last_change":"2024-04-20T02:53:41.994920+0000","last_active":"2024-04-20T02:53:41.994920+0000","last_peered":"2024-04-20T02:53:41.994920+0000","last_clean":"2024-04-20T02:53:41.994920+0000","last_became_active":"2024-04-20T02:52:46.144032+0000","last_became_peered":"2024-04-20T02:52:46.144032+0000","last_unstale":"2024-04-20T02:53:41.994920+0000","last_undegraded":"2024-04-20T02:53:41.994920+0000","last_fullsized":"2024-04-20T02:53:41.994920+0000","mapping_epoch":147,"log_start":"186'285","ondisk_log_start":"186'285","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'288","last_scrub_stamp":"2024-04-20T02:53:41.994768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:41.994768+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":285,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:46.855967+0000","scrub_duration":0.0029328100000000001,"objects_trimmed":0,"snaptrim_duration":0.039094669999999998,"stat_sum":{"num_bytes":8353244,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":439,"num_read_kb":540894,"num_write":164,"num_write_kb":44217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":27793283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"184'124","reported_seq":658,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148057+0000","last_change":"2024-04-20T02:53:32.932202+0000","last_active":"2024-04-20T02:53:41.148057+0000","last_peered":"2024-04-20T02:53:41.148057+0000","last_clean":"2024-04-20T02:53:41.148057+0000","last_became_active":"2024-04-20T02:53:18.149105+0000","last_became_peered":"2024-04-20T02:53:18.149105+0000","last_unstale":"2024-04-20T02:53:41.148057+0000","last_undegraded":"2024-04-20T02:53:41.148057+0000","last_fullsized":"2024-04-20T02:53:41.148057+0000","mapping_epoch":178,"log_start":"173'118","ondisk_log_start":"173'118","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'124","last_scrub_stamp":"2024-04-20T02:53:32.932078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.932078+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":118,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:57.199297+0000","scrub_duration":0.0021487580000000002,"objects_trimmed":0,"snaptrim_duration":0.99770691499999997,"stat_sum":{"num_bytes":3882450,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":196,"num_read_kb":274578,"num_write":59,"num_write_kb":18271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":10957139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"163'44","reported_seq":505,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.951958+0000","last_change":"2024-04-20T02:53:25.198652+0000","last_active":"2024-04-20T02:53:41.951958+0000","last_peered":"2024-04-20T02:53:41.951958+0000","last_clean":"2024-04-20T02:53:41.951958+0000","last_became_active":"2024-04-20T02:52:54.156070+0000","last_became_peered":"2024-04-20T02:52:54.156070+0000","last_unstale":"2024-04-20T02:53:41.951958+0000","last_undegraded":"2024-04-20T02:53:41.951958+0000","last_fullsized":"2024-04-20T02:53:41.951958+0000","mapping_epoch":154,"log_start":"153'40","ondisk_log_start":"153'40","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'36","last_scrub_stamp":"2024-04-20T02:52:46.138400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:46.138400+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":40,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:09.709596+0000","scrub_duration":0.003388247,"objects_trimmed":0,"snaptrim_duration":0.038600218999999998,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":116380,"num_write":27,"num_write_kb":7507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1714357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":120989167,"num_objects":48,"num_object_clones":7,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3077,"num_read_kb":3793692,"num_write":1071,"num_write_kb":295960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":217268250,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":17,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295653376,"data_stored":319204417,"data_compressed":515252,"data_compressed_allocated":434176,"data_compressed_original":1310720,"omap_allocated":2595,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":48,"acting":48,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":66983468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":225505280,"data_stored":240729994,"data_compressed":12805896,"data_compressed_allocated":13840384,"data_compressed_original":29753344,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509487,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76944,"kb_used_data":41732,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94294896,"statfs":{"total":96636764160,"available":96557973504,"internally_reserved":0,"allocated":42733568,"data_stored":45684044,"data_compressed":90764,"data_compressed_allocated":102400,"data_compressed_original":774144,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.44}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.498},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.64400000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.53000000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.435,"15min":0.443},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65500000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.59399999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.441,"15min":0.45000000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57699999999999996,"5min":2.431,"15min":2.431},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.66900000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757283,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103468,"kb_used_data":67544,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94268372,"statfs":{"total":96636764160,"available":96530812928,"internally_reserved":0,"allocated":69165056,"data_stored":72513551,"data_compressed":1987361,"data_compressed_allocated":2129920,"data_compressed_original":4653056,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56599999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42299999999999999,"15min":0.41699999999999998},"min":{"1min":0.34699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.51500000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.47899999999999998,"15min":0.46100000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.437,"15min":0.434},"min":{"1min":0.34499999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.438}]},{"osd":4,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.29699999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.27900000000000003,"5min":0.252,"15min":0.252},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.33700000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.441,"15min":0.437},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.312},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.36899999999999999}]}]},{"osd":2,"up_from":0,"seq":592705486863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116752,"kb_used_data":81540,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94255088,"statfs":{"total":96636764160,"available":96517210112,"internally_reserved":0,"allocated":83496960,"data_stored":87000195,"data_compressed":1830696,"data_compressed_allocated":1974272,"data_compressed_original":4153344,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.28199999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.41799999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.753,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.63600000000000001,"5min":1.097,"15min":1.097},"last":0.621},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.53500000000000003,"15min":0.54900000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.56399999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.54500000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.53200000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.59199999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104584,"kb_used_data":69376,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94267256,"statfs":{"total":96636764160,"available":96529670144,"internally_reserved":0,"allocated":71041024,"data_stored":73629898,"data_compressed":1956129,"data_compressed_allocated":2117632,"data_compressed_original":4587520,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.32300000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.31},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.23200000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.27600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.85899999999999999,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.52900000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.01,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.435},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47199999999999998,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69899999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.36899999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.39200000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.46700000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126684,"kb_used_data":91468,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94245156,"statfs":{"total":96636764160,"available":96507039744,"internally_reserved":0,"allocated":93663232,"data_stored":97384203,"data_compressed":2785685,"data_compressed_allocated":3022848,"data_compressed_original":7024640,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.504}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.432},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.51100000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.625,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.377},{"interface":"front","average":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.495}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.57799999999999996,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.66100000000000003,"5min":1.038,"15min":1.038},"last":0.48199999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.52800000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509483,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119556,"kb_used_data":84336,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94252284,"statfs":{"total":96636764160,"available":96514338816,"internally_reserved":0,"allocated":86360064,"data_stored":89725822,"data_compressed":2819490,"data_compressed_allocated":3063808,"data_compressed_original":6762496,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.435},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.56299999999999994}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90500000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69599999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.55500000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.58499999999999996}]}]},{"osd":4,"up_from":0,"seq":717259538441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49860608,"data_stored":52937332,"data_compressed":2759802,"data_compressed_allocated":2973696,"data_compressed_original":6184960,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31961088,"data_stored":33949988,"data_compressed":1733638,"data_compressed_allocated":1871872,"data_compressed_original":3956736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12894208,"data_stored":13909415,"data_compressed":734140,"data_compressed_allocated":790528,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18509824,"data_stored":19818665,"data_compressed":1037033,"data_compressed_allocated":1118208,"data_compressed_original":2482176,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34250752,"data_stored":35250075,"data_compressed":54370,"data_compressed_allocated":57344,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49352704,"data_stored":52045922,"data_compressed":97058,"data_compressed_allocated":102400,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2595,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36118528,"data_stored":37574963,"data_compressed":11526,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56590336,"data_stored":67896718,"data_compressed":95804,"data_compressed_allocated":53248,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":34230272,"data_stored":36453414,"data_compressed":42758,"data_compressed_allocated":24576,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":44625920,"data_stored":45837696,"data_compressed":128290,"data_compressed_allocated":114688,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40484864,"data_stored":44145629,"data_compressed":85446,"data_compressed_allocated":69632,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:53:47.697 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-20T02:53:47.698 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-04-20T02:53:47.705 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-20T02:53:47.780 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:47.780+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:47.860 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:47.860+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:47.869 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:47.869 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:47.882 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-20T02:53:47.903 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-20T02:53:47.966 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:47.966+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:48.056 INFO:teuthology.orchestra.run.smithi105.stdout:ERROR: (22) Invalid argument 2024-04-20T02:53:48.056 INFO:teuthology.orchestra.run.smithi105.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-20T02:53:48.062 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:48.062+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:48.067 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-20T02:53:48.071 INFO:teuthology.orchestra.run.smithi105.stdout:osd.0: {} 2024-04-20T02:53:48.091 INFO:teuthology.orchestra.run.smithi105.stdout:osd.1: {} 2024-04-20T02:53:48.111 INFO:teuthology.orchestra.run.smithi105.stdout:osd.2: {} 2024-04-20T02:53:48.130 INFO:teuthology.orchestra.run.smithi105.stdout:osd.3: {} 2024-04-20T02:53:48.150 INFO:teuthology.orchestra.run.smithi105.stdout:osd.4: {} 2024-04-20T02:53:48.163 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:48.164+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:48.169 INFO:teuthology.orchestra.run.smithi105.stdout:osd.5: {} 2024-04-20T02:53:48.189 INFO:teuthology.orchestra.run.smithi105.stdout:osd.6: {} 2024-04-20T02:53:48.204 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:48.204 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:48.204 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:48.204 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:48.204 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.204 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1131 4.b 4:d4308fef:::smithi10548823-6:head [write 3126680~419564] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:48.204 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:48.204 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 177.270672289, 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082395336999999999, 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1131 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.205 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932583+0000", 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932582+0000", 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932806+0000", 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0002243 2024-04-20T02:53:48.206 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932821+0000", 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4939e-05 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932834+0000", 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2883e-05 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952712+0000", 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019878759999999999 2024-04-20T02:53:48.207 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952823+0000", 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000111032 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.952996+0000", 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000172762 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966708+0000", 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013711406000000001 2024-04-20T02:53:48.208 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.979996+0000", 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013288413000000001 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014893+0000", 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034896609000000002 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014928+0000", 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5068999999999999e-05 2024-04-20T02:53:48.209 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.014978+0000", 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0138e-05 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1132 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (55) in=7b,truncate 3546244] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 177.270412083, 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082298426999999993, 2024-04-20T02:53:48.210 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1132 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.211 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932843+0000", 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932842+0000", 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.212 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932847+0000", 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4800000000000003e-06 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932854+0000", 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9979999999999997e-06 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932859+0000", 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4559999999999999e-06 2024-04-20T02:53:48.213 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964509+0000", 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031650373000000002 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964614+0000", 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010463699999999999 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:48.214 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.964786+0000", 2024-04-20T02:53:48.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000171752 2024-04-20T02:53:48.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.215 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.966739+0000", 2024-04-20T02:53:48.215 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019530229999999999 2024-04-20T02:53:48.215 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.215 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.215 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.980009+0000", 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0132703 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015037+0000", 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035027871000000002 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015083+0000", 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6526000000000001e-05 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.216 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015141+0000", 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7679000000000001e-05 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1133 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 177.270390615, 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.082453327000000007, 2024-04-20T02:53:48.217 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1133 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.218 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932864+0000", 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.219 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932866+0000", 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6649999999999999e-06 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932874+0000", 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2010000000000002e-06 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.932879+0000", 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2580000000000003e-06 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.220 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965112+0000", 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.032233428000000001 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:50.965202+0000", 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9815000000000007e-05 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015177+0000", 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049974563 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.221 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015235+0000", 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8014999999999999e-05 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:51.015318+0000", 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2818999999999994e-05 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.222 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3771 4.b 4:d4308fef:::smithi10548823-6:head [write 2548663~763022] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.021361958, 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072733780999999997, 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3771 2024-04-20T02:53:48.223 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181893+0000", 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.181891+0000", 2024-04-20T02:53:48.224 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182301+0000", 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00040985099999999998 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182317+0000", 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6305999999999999e-05 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182329+0000", 2024-04-20T02:53:48.225 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1337e-05 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198720+0000", 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016391606 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198807+0000", 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7319999999999997e-05 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:48.226 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.198925+0000", 2024-04-20T02:53:48.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117345 2024-04-20T02:53:48.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.227 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.227 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218062+0000", 2024-04-20T02:53:48.227 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019137134 2024-04-20T02:53:48.227 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.227 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234990+0000", 2024-04-20T02:53:48.228 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016928367 2024-04-20T02:53:48.228 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.228 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.228 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.228 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254488+0000", 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019497460000000001 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254571+0000", 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3029999999999996e-05 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254627+0000", 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5865000000000001e-05 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.229 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3772 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3395047] snapc 43=[43,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.020915385999999, 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072419708999999999, 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3772 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.230 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182339+0000", 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.231 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182343+0000", 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0099999999999997e-06 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182349+0000", 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9569999999999999e-06 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182354+0000", 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.826e-06 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.232 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217228+0000", 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034874555000000002 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217312+0000", 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3478000000000003e-05 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,4", 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217431+0000", 2024-04-20T02:53:48.233 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119512 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.234969+0000", 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017537833999999999 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.235012+0000", 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3411000000000002e-05 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.234 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254638+0000", 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019625815000000001 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254682+0000", 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3945e-05 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254759+0000", 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6882e-05 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.235 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3773 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e141)", 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 69.020896778999997, 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072538680999999994, 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3773 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.236 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182358+0000", 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.237 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182360+0000", 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3309999999999998e-06 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182366+0000", 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.981e-06 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.182371+0000", 2024-04-20T02:53:48.238 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5390000000000001e-06 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.217932+0000", 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035561700000000002 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.218020+0000", 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7352e-05 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254778+0000", 2024-04-20T02:53:48.239 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036757971 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254819+0000", 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1525e-05 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:39.254897+0000", 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7621999999999996e-05 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.240 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4057 4.b 4:d4308fef:::smithi10548823-6:head [write 749678~790384,stat] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.060042039000002, 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074563737000000005, 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4057 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.241 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143213+0000", 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143210+0000", 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.242 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143760+0000", 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00055015999999999995 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143772+0000", 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1345000000000001e-05 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143781+0000", 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0929999999999992e-06 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.243 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143895+0000", 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113998 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143906+0000", 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1195999999999999e-05 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.153590+0000", 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0096836000000000005 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.244 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155611+0000", 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.002021012 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217691+0000", 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062080244 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217777+0000", 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5583999999999999e-05 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.245 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4058 4.b 4:d4308fef:::smithi10548823-6:head [write 2332155~504118] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.059461013000004, 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074090323999999999, 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4058 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.246 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143794+0000", 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143793+0000", 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.247 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144104+0000", 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000310971 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144112+0000", 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1990000000000006e-06 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144118+0000", 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.443e-06 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.248 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144162+0000", 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3066999999999999e-05 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144172+0000", 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0083e-05 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155911+0000", 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011739657000000001 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.249 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.155993+0000", 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1754000000000004e-05 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217810+0000", 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061816958999999998 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217884+0000", 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4098999999999996e-05 2024-04-20T02:53:48.250 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4059 4.b 4:d4308fef:::smithi10548823-6:head [write 3271857~549469] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.059124851999997, 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073860173000000001, 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.251 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4059 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144130+0000", 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.252 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144129+0000", 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144439+0000", 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00030924000000000001 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144447+0000", 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8930000000000005e-06 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.253 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144455+0000", 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1410000000000005e-06 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144505+0000", 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0698000000000001e-05 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144520+0000", 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4605e-05 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.254 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156034+0000", 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011513809 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156130+0000", 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6039999999999995e-05 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217915+0000", 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061785501999999999 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.255 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217990+0000", 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4770000000000004e-05 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4060 4.b 4:d4308fef:::smithi10548823-6:head [setxattr _header (56) in=7b,truncate 3949678] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.058789472999997, 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085188106999999999, 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.256 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4060 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:48.257 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144465+0000", 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144469+0000", 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.946e-06 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.258 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144476+0000", 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7909999999999999e-06 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144482+0000", 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.601e-06 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144803+0000", 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00032103699999999998 2024-04-20T02:53:48.259 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144813+0000", 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0183999999999999e-05 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156168+0000", 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011354523 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156254+0000", 2024-04-20T02:53:48.260 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5706999999999998e-05 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218023+0000", 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.061769566999999997 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218137+0000", 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000113912 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,3", 2024-04-20T02:53:48.261 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218377+0000", 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023941000000000001 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225107+0000", 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0067304510000000001 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225121+0000", 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4341e-05 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.262 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229481+0000", 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0043599880000000004 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229522+0000", 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0224e-05 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229653+0000", 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013190500000000001 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.263 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4061 4.b 4:d4308fef:::smithi10548823-6:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.058649856000002, 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.085431572999999997, 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4061 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.264 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144605+0000", 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144604+0000", 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.265 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144608+0000", 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1080000000000003e-06 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144618+0000", 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.5149999999999995e-06 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144625+0000", 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8260000000000004e-06 2024-04-20T02:53:48.266 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144846+0000", 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022160699999999999 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144854+0000", 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0399999999999993e-06 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156327+0000", 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011472178 2024-04-20T02:53:48.267 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for degraded object", 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156457+0000", 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013012599999999999 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218704+0000", 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062247080000000003 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.218795+0000", 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1391000000000006e-05 2024-04-20T02:53:48.268 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229703+0000", 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010907834999999999 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.229818+0000", 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000115423 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.230037+0000", 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021803600000000001 2024-04-20T02:53:48.269 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4146 3.1 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head [write 0~1129472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.023053879000003, 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.067860845000000003, 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.270 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4146 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180201+0000", 2024-04-20T02:53:48.271 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180199+0000", 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180439+0000", 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000239375 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.272 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180449+0000", 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0220999999999999e-05 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.180456+0000", 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0090000000000001e-06 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.273 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181170+0000", 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00071358900000000004 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181331+0000", 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016178500000000001 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,5", 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181490+0000", 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000158703 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.274 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225260+0000", 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.043770166999999999 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226398+0000", 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001138189 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247971+0000", 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021572728999999999 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.275 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248021+0000", 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0176999999999997e-05 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.248062+0000", 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0472000000000001e-05 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4623 4.3 4:c07c6daa:::smithi10548823-7:head [write 2772312~34523] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:48.276 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.053126031000001, 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074418926999999996, 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4623 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.277 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150129+0000", 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150128+0000", 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150143+0000", 2024-04-20T02:53:48.278 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5166e-05 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150147+0000", 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.924e-06 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150151+0000", 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.777e-06 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.279 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150180+0000", 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8472000000000001e-05 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150184+0000", 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4349999999999999e-06 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187063+0000", 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036878474000000001 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.280 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187145+0000", 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2107999999999994e-05 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.187280+0000", 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135384 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189223+0000", 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0019430820000000001 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.281 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189411+0000", 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000188061 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224429+0000", 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035017386999999997 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224487+0000", 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8884000000000001e-05 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.282 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224548+0000", 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0238000000000002e-05 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4624 4.3 4:c07c6daa:::smithi10548823-7:head [setxattr _header (56) in=7b,truncate 2806835] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.053099786000001, 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074609524999999996, 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.283 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4624 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.284 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150155+0000", 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150158+0000", 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9799999999999998e-06 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150161+0000", 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8140000000000001e-06 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.285 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150164+0000", 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5329999999999999e-06 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150197+0000", 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2926999999999998e-05 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150201+0000", 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2989999999999998e-06 2024-04-20T02:53:48.286 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188464+0000", 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.038262276999999997 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188576+0000", 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112413 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.188756+0000", 2024-04-20T02:53:48.287 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000180206 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189459+0000", 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00070307500000000005 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224557+0000", 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035097721999999998 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224633+0000", 2024-04-20T02:53:48.288 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5798999999999997e-05 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224690+0000", 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6784999999999999e-05 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224765+0000", 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4969999999999995e-05 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.289 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4625 4.3 4:c07c6daa:::smithi10548823-7:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.053087823999999, 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074770387999999993, 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4625 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.290 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150167+0000", 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.291 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150169+0000", 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7680000000000001e-06 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150171+0000", 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8779999999999998e-06 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150175+0000", 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2909999999999999e-06 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.292 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150215+0000", 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0074000000000003e-05 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150219+0000", 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.177e-06 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189474+0000", 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039255274999999999 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.293 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189531+0000", 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7188999999999997e-05 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224802+0000", 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035270981999999999 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224861+0000", 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8276999999999998e-05 2024-04-20T02:53:48.294 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224937+0000", 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6689999999999999e-05 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4661 4.3 4:cd374330:::smithi10548823-1:head [write 490870~618799] snapc 5c=[5c] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.053077066, 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074896037999999998, 2024-04-20T02:53:48.295 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4661 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.296 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150178+0000", 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150452+0000", 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00027429199999999998 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.297 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150458+0000", 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5810000000000002e-06 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150463+0000", 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7029999999999997e-06 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150486+0000", 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3308000000000001e-05 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.298 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150491+0000", 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6179999999999997e-06 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189565+0000", 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.039073926000000002 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189713+0000", 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014765700000000001 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.299 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.189930+0000", 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021749700000000001 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224591+0000", 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034661210999999997 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224775+0000", 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000184091 2024-04-20T02:53:48.300 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224987+0000", 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021128100000000001 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225040+0000", 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3395e-05 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225074+0000", 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3699999999999999e-05 2024-04-20T02:53:48.301 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4662 4.3 4:cd374330:::smithi10548823-1:head [write 1592676~774144] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.052786181999998, 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075836991000000006, 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.302 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4662 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150469+0000", 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.303 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150468+0000", 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150808+0000", 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033972899999999999 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150815+0000", 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8159999999999996e-06 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.304 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150819+0000", 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150855+0000", 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150886+0000", 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.0876000000000001e-05 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.305 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.204837+0000", 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053950873000000003 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.204916+0000", 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9008e-05 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.205039+0000", 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012294000000000001 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.306 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.224780+0000", 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.019741771000000002 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225703+0000", 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00092290600000000003 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226196+0000", 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049284199999999995 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.307 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226250+0000", 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4413000000000002e-05 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226306+0000", 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.516e-05 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4663 4.3 4:cd374330:::smithi10548823-1:head [write 2793074~97796] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:48.308 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.052430609000002, 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075759200999999998, 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4663 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.309 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150824+0000", 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150883+0000", 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9308999999999997e-05 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.310 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150888+0000", 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1139999999999997e-06 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150892+0000", 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5899999999999999e-06 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150924+0000", 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2289000000000002e-05 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.311 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150929+0000", 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0259999999999998e-06 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221580+0000", 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070650828999999998 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221708+0000", 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012845700000000001 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.312 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.221897+0000", 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000188789 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225650+0000", 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0037528980000000002 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226379+0000", 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00072901200000000002 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.313 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226492+0000", 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112905 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226526+0000", 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711000000000003e-05 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226583+0000", 2024-04-20T02:53:48.314 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7624999999999998e-05 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4664 4.3 4:cd374330:::smithi10548823-1:head [setxattr _header (56) in=7b,truncate 2890870] snapc 5c=[5c] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.052356978999999, 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076283431999999998, 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.315 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4664 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.316 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150898+0000", 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150900+0000", 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.571e-06 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150903+0000", 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9409999999999999e-06 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.317 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150907+0000", 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5599999999999998e-06 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150988+0000", 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1544000000000004e-05 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151006+0000", 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.7754999999999999e-05 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.318 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225114+0000", 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074107889999999996 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225189+0000", 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4851999999999994e-05 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,7", 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225305+0000", 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011592899999999999 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.319 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225800+0000", 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00049498200000000004 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.226624+0000", 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00082437000000000005 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227055+0000", 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00043119499999999999 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.320 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227113+0000", 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.7500999999999997e-05 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227181+0000", 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8625000000000001e-05 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.321 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4665 4.3 4:cd374330:::smithi10548823-1:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 27.052332582999998, 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076413855000000003, 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4665 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:48.322 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150922+0000", 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150924+0000", 2024-04-20T02:53:48.323 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8190000000000001e-06 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150929+0000", 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6400000000000002e-06 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.150933+0000", 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5039999999999998e-06 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151025+0000", 2024-04-20T02:53:48.324 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2354000000000004e-05 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.151040+0000", 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4708999999999999e-05 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225553+0000", 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074513356000000003 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.225640+0000", 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.6335000000000006e-05 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227206+0000", 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001566448 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227261+0000", 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4907000000000001e-05 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.326 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:53:21.227336+0000", 2024-04-20T02:53:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.5050999999999997e-05 2024-04-20T02:53:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.327 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.327 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:48.329 INFO:teuthology.orchestra.run.smithi105.stdout:osd.7: {} 2024-04-20T02:53:48.330 INFO:teuthology.orchestra.run.smithi105.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-20T02:53:48.330 INFO:teuthology.orchestra.run.smithi105.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-20T02:53:48.330 INFO:teuthology.orchestra.run.smithi105.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-20T02:53:48.330 INFO:teuthology.orchestra.run.smithi105.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-20T02:53:48.330 INFO:teuthology.orchestra.run.smithi105.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-20T02:53:48.331 INFO:teuthology.orchestra.run.smithi105.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-20T02:53:48.331 INFO:teuthology.orchestra.run.smithi105.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-20T02:53:48.331 INFO:teuthology.orchestra.run.smithi105.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-20T02:53:48.332 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-20T02:53:48.333 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:48.328+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:48.361 INFO:teuthology.orchestra.run.smithi105.stderr:set pool 4 pg_num to 26 2024-04-20T02:53:48.368 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-20T02:53:48.429 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:48.429+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:48.453 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:53:48.497 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:48.498 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:48.498 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:48.509 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-20T02:53:48.550 INFO:tasks.ceph.osd.1.smithi105.stderr:2024-04-20T02:53:48.550+0000 7ff2b60f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 43297) UID: 0 2024-04-20T02:53:48.630 INFO:tasks.ceph.osd.3.smithi105.stderr:2024-04-20T02:53:48.630+0000 7f27645e4700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 43301) UID: 0 2024-04-20T02:53:48.682 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:48.682 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:48.682 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:48.692 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-20T02:53:48.734 INFO:tasks.ceph.osd.0.smithi105.stderr:2024-04-20T02:53:48.734+0000 7f07b1393700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 43294) UID: 0 2024-04-20T02:53:48.803 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:53:48.805 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:53:48.817 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-04-20T02:53:48.357012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":219655603,"num_objects":178,"num_object_clones":7,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":4128,"num_read_kb":3828471,"num_write":2451,"num_write_kb":622342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":284251718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":17,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":648112,"kb_used_data":436116,"kb_used_omap":9,"kb_used_meta":211894,"kb_avail":565582928,"statfs":{"total":579820584960,"available":579156918272,"internally_reserved":0,"allocated":446582784,"data_stored":465989851,"data_compressed":11470125,"data_compressed_allocated":12410880,"data_compressed_original":27955200,"omap_allocated":9619,"internal_metadata":216980077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.025068"},"pg_stats":[{"pgid":"4.1","version":"184'139","reported_seq":711,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148523+0000","last_change":"2024-04-20T02:53:29.153095+0000","last_active":"2024-04-20T02:53:41.148523+0000","last_peered":"2024-04-20T02:53:41.148523+0000","last_clean":"2024-04-20T02:53:41.148523+0000","last_became_active":"2024-04-20T02:53:29.152688+0000","last_became_peered":"2024-04-20T02:53:29.152688+0000","last_unstale":"2024-04-20T02:53:41.148523+0000","last_undegraded":"2024-04-20T02:53:41.148523+0000","last_fullsized":"2024-04-20T02:53:41.148523+0000","mapping_epoch":154,"log_start":"184'139","ondisk_log_start":"184'139","created":22,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'131","last_scrub_stamp":"2024-04-20T02:52:54.612743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.612743+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":368,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:59.591499+0000","scrub_duration":0.0022862500000000001,"objects_trimmed":1,"snaptrim_duration":0.011020622000000001,"stat_sum":{"num_bytes":8084332,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":252,"num_read_kb":318196,"num_write":63,"num_write_kb":15810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":28106829,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":158,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-20T02:53:45.438678+0000","last_change":"2024-04-20T02:53:30.912852+0000","last_active":"2024-04-20T02:53:45.438678+0000","last_peered":"2024-04-20T02:53:45.438678+0000","last_clean":"2024-04-20T02:53:45.438678+0000","last_became_active":"2024-04-20T02:53:18.152172+0000","last_became_peered":"2024-04-20T02:53:18.152172+0000","last_unstale":"2024-04-20T02:53:45.438678+0000","last_undegraded":"2024-04-20T02:53:45.438678+0000","last_fullsized":"2024-04-20T02:53:45.438678+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:32.689737+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"186'205","reported_seq":819,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-20T02:53:45.347860+0000","last_change":"2024-04-20T02:53:33.343239+0000","last_active":"2024-04-20T02:53:45.347860+0000","last_peered":"2024-04-20T02:53:45.347860+0000","last_clean":"2024-04-20T02:53:45.347860+0000","last_became_active":"2024-04-20T02:53:33.159428+0000","last_became_peered":"2024-04-20T02:53:33.159428+0000","last_unstale":"2024-04-20T02:53:45.347860+0000","last_undegraded":"2024-04-20T02:53:45.347860+0000","last_fullsized":"2024-04-20T02:53:45.347860+0000","mapping_epoch":176,"log_start":"186'205","ondisk_log_start":"186'205","created":22,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'205","last_scrub_stamp":"2024-04-20T02:53:33.342996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:33.342996+0000","objects_scrubbed":4,"log_size":0,"log_dups_size":343,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:49.442548+0000","scrub_duration":0.0033447149999999998,"objects_trimmed":0,"snaptrim_duration":0.042836615000000001,"stat_sum":{"num_bytes":9545632,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":378,"num_read_kb":480412,"num_write":113,"num_write_kb":33005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33476197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":226,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-04-20T02:53:43.334643+0000","last_change":"2024-04-20T02:53:19.107136+0000","last_active":"2024-04-20T02:53:43.334643+0000","last_peered":"2024-04-20T02:53:43.334643+0000","last_clean":"2024-04-20T02:53:43.334643+0000","last_became_active":"2024-04-20T02:52:51.153987+0000","last_became_peered":"2024-04-20T02:52:51.153987+0000","last_unstale":"2024-04-20T02:53:43.334643+0000","last_undegraded":"2024-04-20T02:53:43.334643+0000","last_fullsized":"2024-04-20T02:53:43.334643+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:24.234001+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":224,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688545+0000","last_change":"2024-04-20T02:52:53.937048+0000","last_active":"2024-04-20T02:53:42.688545+0000","last_peered":"2024-04-20T02:53:42.688545+0000","last_clean":"2024-04-20T02:53:42.688545+0000","last_became_active":"2024-04-20T02:52:51.154174+0000","last_became_peered":"2024-04-20T02:52:51.154174+0000","last_unstale":"2024-04-20T02:53:42.688545+0000","last_undegraded":"2024-04-20T02:53:42.688545+0000","last_fullsized":"2024-04-20T02:53:42.688545+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:53.936957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:53.936957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.656218+0000","scrub_duration":0.0013534090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"187'315","reported_seq":886,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688554+0000","last_change":"2024-04-20T02:53:23.159920+0000","last_active":"2024-04-20T02:53:42.688554+0000","last_peered":"2024-04-20T02:53:42.688554+0000","last_clean":"2024-04-20T02:53:42.688554+0000","last_became_active":"2024-04-20T02:53:21.153338+0000","last_became_peered":"2024-04-20T02:53:21.153338+0000","last_unstale":"2024-04-20T02:53:42.688554+0000","last_undegraded":"2024-04-20T02:53:42.688554+0000","last_fullsized":"2024-04-20T02:53:42.688554+0000","mapping_epoch":47,"log_start":"186'312","ondisk_log_start":"186'312","created":22,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'243","last_scrub_stamp":"2024-04-20T02:52:54.597931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:54.597931+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":557,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:53:59.861555+0000","scrub_duration":0.0028458709999999998,"objects_trimmed":2,"snaptrim_duration":0.0095485269999999994,"stat_sum":{"num_bytes":15467220,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":413,"num_read_kb":563626,"num_write":89,"num_write_kb":25274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":19243742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":620,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688396+0000","last_change":"2024-04-20T02:53:08.766997+0000","last_active":"2024-04-20T02:53:42.688396+0000","last_peered":"2024-04-20T02:53:42.688396+0000","last_clean":"2024-04-20T02:53:42.688396+0000","last_became_active":"2024-04-20T02:52:52.148432+0000","last_became_peered":"2024-04-20T02:52:52.148432+0000","last_unstale":"2024-04-20T02:53:42.688396+0000","last_undegraded":"2024-04-20T02:53:42.688396+0000","last_fullsized":"2024-04-20T02:53:42.688396+0000","mapping_epoch":152,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:29.132762+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":99,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688379+0000","last_change":"2024-04-20T02:52:32.918930+0000","last_active":"2024-04-20T02:53:42.688379+0000","last_peered":"2024-04-20T02:53:42.688379+0000","last_clean":"2024-04-20T02:53:42.688379+0000","last_became_active":"2024-04-20T02:52:22.261878+0000","last_became_peered":"2024-04-20T02:52:22.261878+0000","last_unstale":"2024-04-20T02:53:42.688379+0000","last_undegraded":"2024-04-20T02:53:42.688379+0000","last_fullsized":"2024-04-20T02:53:42.688379+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:32.918813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.918813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014288059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"186'153","reported_seq":744,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-04-20T02:53:43.334596+0000","last_change":"2024-04-20T02:53:31.855215+0000","last_active":"2024-04-20T02:53:43.334596+0000","last_peered":"2024-04-20T02:53:43.334596+0000","last_clean":"2024-04-20T02:53:43.334596+0000","last_became_active":"2024-04-20T02:53:16.151498+0000","last_became_peered":"2024-04-20T02:53:16.151498+0000","last_unstale":"2024-04-20T02:53:43.334596+0000","last_undegraded":"2024-04-20T02:53:43.334596+0000","last_fullsized":"2024-04-20T02:53:43.334596+0000","mapping_epoch":176,"log_start":"185'149","ondisk_log_start":"185'149","created":22,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'153","last_scrub_stamp":"2024-04-20T02:53:31.855077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:31.855077+0000","objects_scrubbed":5,"log_size":4,"log_dups_size":236,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.406736+0000","scrub_duration":0.0052908479999999999,"objects_trimmed":0,"snaptrim_duration":0.041643649999999997,"stat_sum":{"num_bytes":9717018,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":174,"num_read_kb":188998,"num_write":46,"num_write_kb":11974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22843153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":619,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-20T02:53:45.348148+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:53:45.348148+0000","last_peered":"2024-04-20T02:53:45.348148+0000","last_clean":"2024-04-20T02:53:45.348148+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:53:45.348148+0000","last_undegraded":"2024-04-20T02:53:45.348148+0000","last_fullsized":"2024-04-20T02:53:45.348148+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":367,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688477+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:53:42.688477+0000","last_peered":"2024-04-20T02:53:42.688477+0000","last_clean":"2024-04-20T02:53:42.688477+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:53:42.688477+0000","last_undegraded":"2024-04-20T02:53:42.688477+0000","last_fullsized":"2024-04-20T02:53:42.688477+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":200,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148216+0000","last_change":"2024-04-20T02:53:08.364116+0000","last_active":"2024-04-20T02:53:41.148216+0000","last_peered":"2024-04-20T02:53:41.148216+0000","last_clean":"2024-04-20T02:53:41.148216+0000","last_became_active":"2024-04-20T02:52:22.265487+0000","last_became_peered":"2024-04-20T02:52:22.265487+0000","last_unstale":"2024-04-20T02:53:41.148216+0000","last_undegraded":"2024-04-20T02:53:41.148216+0000","last_fullsized":"2024-04-20T02:53:41.148216+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.792324+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"186'105","reported_seq":375,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-20T02:53:45.438651+0000","last_change":"2024-04-20T02:53:25.163140+0000","last_active":"2024-04-20T02:53:45.438651+0000","last_peered":"2024-04-20T02:53:45.438651+0000","last_clean":"2024-04-20T02:53:45.438651+0000","last_became_active":"2024-04-20T02:53:18.152319+0000","last_became_peered":"2024-04-20T02:53:18.152319+0000","last_unstale":"2024-04-20T02:53:45.438651+0000","last_undegraded":"2024-04-20T02:53:45.438651+0000","last_fullsized":"2024-04-20T02:53:45.438651+0000","mapping_epoch":178,"log_start":"184'101","ondisk_log_start":"184'101","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'86","last_scrub_stamp":"2024-04-20T02:52:32.901998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:32.901998+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":131,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024332220000000001,"objects_trimmed":0,"snaptrim_duration":0.99220328899999999,"stat_sum":{"num_bytes":5395179,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":35504,"num_write":16,"num_write_kb":3789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1543958,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":547,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-04-20T02:53:43.333918+0000","last_change":"2024-04-20T02:52:42.201016+0000","last_active":"2024-04-20T02:53:43.333918+0000","last_peered":"2024-04-20T02:53:43.333918+0000","last_clean":"2024-04-20T02:53:43.333918+0000","last_became_active":"2024-04-20T02:52:25.272853+0000","last_became_peered":"2024-04-20T02:52:25.272853+0000","last_unstale":"2024-04-20T02:53:43.333918+0000","last_undegraded":"2024-04-20T02:53:43.333918+0000","last_fullsized":"2024-04-20T02:53:43.333918+0000","mapping_epoch":126,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'188","last_scrub_stamp":"2024-04-20T02:52:42.200865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:52:42.200865+0000","objects_scrubbed":31,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:05.320740+0000","scrub_duration":0.0052191850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":15696807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":105,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-20T02:53:45.348160+0000","last_change":"2024-04-20T02:52:56.212555+0000","last_active":"2024-04-20T02:53:45.348160+0000","last_peered":"2024-04-20T02:53:45.348160+0000","last_clean":"2024-04-20T02:53:45.348160+0000","last_became_active":"2024-04-20T02:52:22.272822+0000","last_became_peered":"2024-04-20T02:52:22.272822+0000","last_unstale":"2024-04-20T02:53:45.348160+0000","last_undegraded":"2024-04-20T02:53:45.348160+0000","last_fullsized":"2024-04-20T02:53:45.348160+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:56.212430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.212430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:20.820102+0000","scrub_duration":0.0013799870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"184'130","reported_seq":728,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148502+0000","last_change":"2024-04-20T02:53:23.161348+0000","last_active":"2024-04-20T02:53:41.148502+0000","last_peered":"2024-04-20T02:53:41.148502+0000","last_clean":"2024-04-20T02:53:41.148502+0000","last_became_active":"2024-04-20T02:52:46.141983+0000","last_became_peered":"2024-04-20T02:52:46.141983+0000","last_unstale":"2024-04-20T02:53:41.148502+0000","last_undegraded":"2024-04-20T02:53:41.148502+0000","last_fullsized":"2024-04-20T02:53:41.148502+0000","mapping_epoch":147,"log_start":"179'124","ondisk_log_start":"179'124","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'119","last_scrub_stamp":"2024-04-20T02:52:56.564632+0000","last_deep_scrub":"75'50","last_deep_scrub_stamp":"2024-04-20T02:51:32.487770+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.564632+0000","objects_scrubbed":4,"log_size":6,"log_dups_size":310,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:22.552034+0000","scrub_duration":0.0028929759999999998,"objects_trimmed":1,"snaptrim_duration":0.010818365,"stat_sum":{"num_bytes":2840449,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":43301,"num_write":4,"num_write_kb":1274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":561,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688486+0000","last_change":"2024-04-20T02:53:18.737801+0000","last_active":"2024-04-20T02:53:42.688486+0000","last_peered":"2024-04-20T02:53:42.688486+0000","last_clean":"2024-04-20T02:53:42.688486+0000","last_became_active":"2024-04-20T02:53:11.150822+0000","last_became_peered":"2024-04-20T02:53:11.150822+0000","last_unstale":"2024-04-20T02:53:42.688486+0000","last_undegraded":"2024-04-20T02:53:42.688486+0000","last_fullsized":"2024-04-20T02:53:42.688486+0000","mapping_epoch":171,"log_start":"175'232","ondisk_log_start":"175'232","created":21,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":238,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:19.609708+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":18551655,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-04-20T02:53:43.495007+0000","last_change":"2024-04-20T02:53:43.494597+0000","last_active":"2024-04-20T02:53:43.495007+0000","last_peered":"2024-04-20T02:53:43.495007+0000","last_clean":"2024-04-20T02:53:43.495007+0000","last_became_active":"2024-04-20T02:53:43.494140+0000","last_became_peered":"2024-04-20T02:53:43.494140+0000","last_unstale":"2024-04-20T02:53:43.495007+0000","last_undegraded":"2024-04-20T02:53:43.495007+0000","last_fullsized":"2024-04-20T02:53:43.495007+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:00.725766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"185'48","reported_seq":531,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-04-20T02:53:43.334368+0000","last_change":"2024-04-20T02:53:33.880323+0000","last_active":"2024-04-20T02:53:43.334368+0000","last_peered":"2024-04-20T02:53:43.334368+0000","last_clean":"2024-04-20T02:53:43.334368+0000","last_became_active":"2024-04-20T02:53:17.156757+0000","last_became_peered":"2024-04-20T02:53:17.156757+0000","last_unstale":"2024-04-20T02:53:43.334368+0000","last_undegraded":"2024-04-20T02:53:43.334368+0000","last_fullsized":"2024-04-20T02:53:43.334368+0000","mapping_epoch":141,"log_start":"169'45","ondisk_log_start":"169'45","created":22,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'48","last_scrub_stamp":"2024-04-20T02:53:33.880214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:33.880214+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:59.742146+0000","scrub_duration":0.0018263730000000001,"objects_trimmed":0,"snaptrim_duration":0.039989539999999997,"stat_sum":{"num_bytes":2991712,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":274,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148814+0000","last_change":"2024-04-20T02:53:25.404436+0000","last_active":"2024-04-20T02:53:41.148814+0000","last_peered":"2024-04-20T02:53:41.148814+0000","last_clean":"2024-04-20T02:53:41.148814+0000","last_became_active":"2024-04-20T02:53:25.402753+0000","last_became_peered":"2024-04-20T02:53:25.402753+0000","last_unstale":"2024-04-20T02:53:41.148814+0000","last_undegraded":"2024-04-20T02:53:41.148814+0000","last_fullsized":"2024-04-20T02:53:41.148814+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:52:43.686760+0000","last_clean_scrub_stamp":"2024-04-20T02:52:43.686760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.573670+0000","scrub_duration":0.0018670150000000001,"objects_trimmed":0,"snaptrim_duration":0.001125371,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":217,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-04-20T02:53:43.334284+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:53:43.334284+0000","last_peered":"2024-04-20T02:53:43.334284+0000","last_clean":"2024-04-20T02:53:43.334284+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:53:43.334284+0000","last_undegraded":"2024-04-20T02:53:43.334284+0000","last_fullsized":"2024-04-20T02:53:43.334284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"186'174","reported_seq":699,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-04-20T02:53:41.148201+0000","last_change":"2024-04-20T02:53:33.417307+0000","last_active":"2024-04-20T02:53:41.148201+0000","last_peered":"2024-04-20T02:53:41.148201+0000","last_clean":"2024-04-20T02:53:41.148201+0000","last_became_active":"2024-04-20T02:52:13.145577+0000","last_became_peered":"2024-04-20T02:52:13.145577+0000","last_unstale":"2024-04-20T02:53:41.148201+0000","last_undegraded":"2024-04-20T02:53:41.148201+0000","last_fullsized":"2024-04-20T02:53:41.148201+0000","mapping_epoch":62,"log_start":"186'170","ondisk_log_start":"186'170","created":22,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'174","last_scrub_stamp":"2024-04-20T02:53:33.417175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:33.417175+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":328,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:39.243430+0000","scrub_duration":0.0025256810000000001,"objects_trimmed":0,"snaptrim_duration":0.00066820099999999999,"stat_sum":{"num_bytes":10729969,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":90,"num_read_kb":103553,"num_write":46,"num_write_kb":12413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"195'513","reported_seq":1225,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-04-20T02:53:43.334017+0000","last_change":"2024-04-20T02:53:37.203422+0000","last_active":"2024-04-20T02:53:43.334017+0000","last_peered":"2024-04-20T02:53:43.334017+0000","last_clean":"2024-04-20T02:53:43.334017+0000","last_became_active":"2024-04-20T02:53:11.151224+0000","last_became_peered":"2024-04-20T02:53:11.151224+0000","last_unstale":"2024-04-20T02:53:43.334017+0000","last_undegraded":"2024-04-20T02:53:43.334017+0000","last_fullsized":"2024-04-20T02:53:43.334017+0000","mapping_epoch":171,"log_start":"194'509","ondisk_log_start":"194'509","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'294","last_scrub_stamp":"2024-04-20T02:52:16.813424+0000","last_deep_scrub":"98'280","last_deep_scrub_stamp":"2024-04-20T02:51:56.947305+0000","last_clean_scrub_stamp":"2024-04-20T02:52:16.813424+0000","objects_scrubbed":7,"log_size":4,"log_dups_size":1130,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034018329999999999,"objects_trimmed":0,"snaptrim_duration":1.0562096670000001,"stat_sum":{"num_bytes":18824302,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":223,"num_read_kb":242421,"num_write":112,"num_write_kb":32076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":34386920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"184'120","reported_seq":619,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-20T02:53:45.438492+0000","last_change":"2024-04-20T02:53:44.083863+0000","last_active":"2024-04-20T02:53:45.438492+0000","last_peered":"2024-04-20T02:53:45.438492+0000","last_clean":"2024-04-20T02:53:45.438492+0000","last_became_active":"2024-04-20T02:53:18.153426+0000","last_became_peered":"2024-04-20T02:53:18.153426+0000","last_unstale":"2024-04-20T02:53:45.438492+0000","last_undegraded":"2024-04-20T02:53:45.438492+0000","last_fullsized":"2024-04-20T02:53:45.438492+0000","mapping_epoch":178,"log_start":"177'116","ondisk_log_start":"177'116","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'120","last_scrub_stamp":"2024-04-20T02:53:44.083744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.083744+0000","objects_scrubbed":3,"log_size":4,"log_dups_size":116,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:59.005224+0000","scrub_duration":0.0037127990000000001,"objects_trimmed":0,"snaptrim_duration":0.00042807000000000001,"stat_sum":{"num_bytes":6343443,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":204,"num_read_kb":230945,"num_write":64,"num_write_kb":16950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"184'350","reported_seq":933,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-04-20T02:53:42.688396+0000","last_change":"2024-04-20T02:53:30.856293+0000","last_active":"2024-04-20T02:53:42.688396+0000","last_peered":"2024-04-20T02:53:42.688396+0000","last_clean":"2024-04-20T02:53:42.688396+0000","last_became_active":"2024-04-20T02:53:11.147630+0000","last_became_peered":"2024-04-20T02:53:11.147630+0000","last_unstale":"2024-04-20T02:53:42.688396+0000","last_undegraded":"2024-04-20T02:53:42.688396+0000","last_fullsized":"2024-04-20T02:53:42.688396+0000","mapping_epoch":171,"log_start":"179'344","ondisk_log_start":"179'344","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'350","last_scrub_stamp":"2024-04-20T02:53:30.856060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.856060+0000","objects_scrubbed":5,"log_size":6,"log_dups_size":344,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.470953+0000","scrub_duration":0.0031057490000000001,"objects_trimmed":1,"snaptrim_duration":0.0080676189999999998,"stat_sum":{"num_bytes":11923149,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":388,"num_read_kb":461817,"num_write":178,"num_write_kb":48621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":23150965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"178'159","reported_seq":670,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-20T02:53:45.348302+0000","last_change":"2024-04-20T02:53:42.210611+0000","last_active":"2024-04-20T02:53:45.348302+0000","last_peered":"2024-04-20T02:53:45.348302+0000","last_clean":"2024-04-20T02:53:45.348302+0000","last_became_active":"2024-04-20T02:53:11.148476+0000","last_became_peered":"2024-04-20T02:53:11.148476+0000","last_unstale":"2024-04-20T02:53:45.348302+0000","last_undegraded":"2024-04-20T02:53:45.348302+0000","last_fullsized":"2024-04-20T02:53:45.348302+0000","mapping_epoch":171,"log_start":"173'155","ondisk_log_start":"173'155","created":22,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"178'159","last_scrub_stamp":"2024-04-20T02:53:42.210503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.210503+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":155,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:58.448617+0000","scrub_duration":0.0024829349999999999,"objects_trimmed":0,"snaptrim_duration":0.010578568,"stat_sum":{"num_bytes":4773320,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":178,"num_read_kb":193067,"num_write":90,"num_write_kb":24779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":14051707,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"188'288","reported_seq":945,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-04-20T02:53:43.334519+0000","last_change":"2024-04-20T02:53:41.994920+0000","last_active":"2024-04-20T02:53:43.334519+0000","last_peered":"2024-04-20T02:53:43.334519+0000","last_clean":"2024-04-20T02:53:43.334519+0000","last_became_active":"2024-04-20T02:52:46.144032+0000","last_became_peered":"2024-04-20T02:52:46.144032+0000","last_unstale":"2024-04-20T02:53:43.334519+0000","last_undegraded":"2024-04-20T02:53:43.334519+0000","last_fullsized":"2024-04-20T02:53:43.334519+0000","mapping_epoch":147,"log_start":"186'285","ondisk_log_start":"186'285","created":22,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'288","last_scrub_stamp":"2024-04-20T02:53:41.994768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:41.994768+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":285,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:46.855967+0000","scrub_duration":0.0029328100000000001,"objects_trimmed":0,"snaptrim_duration":0.039094669999999998,"stat_sum":{"num_bytes":8353244,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":439,"num_read_kb":540894,"num_write":164,"num_write_kb":44217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":27793283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"184'124","reported_seq":661,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-04-20T02:53:45.438669+0000","last_change":"2024-04-20T02:53:32.932202+0000","last_active":"2024-04-20T02:53:45.438669+0000","last_peered":"2024-04-20T02:53:45.438669+0000","last_clean":"2024-04-20T02:53:45.438669+0000","last_became_active":"2024-04-20T02:53:18.149105+0000","last_became_peered":"2024-04-20T02:53:18.149105+0000","last_unstale":"2024-04-20T02:53:45.438669+0000","last_undegraded":"2024-04-20T02:53:45.438669+0000","last_fullsized":"2024-04-20T02:53:45.438669+0000","mapping_epoch":178,"log_start":"173'118","ondisk_log_start":"173'118","created":22,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'124","last_scrub_stamp":"2024-04-20T02:53:32.932078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.932078+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":118,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:57.199297+0000","scrub_duration":0.0021487580000000002,"objects_trimmed":0,"snaptrim_duration":0.99770691499999997,"stat_sum":{"num_bytes":3882450,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":196,"num_read_kb":274578,"num_write":59,"num_write_kb":18271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":10957139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"163'44","reported_seq":507,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-04-20T02:53:43.334730+0000","last_change":"2024-04-20T02:53:25.198652+0000","last_active":"2024-04-20T02:53:43.334730+0000","last_peered":"2024-04-20T02:53:43.334730+0000","last_clean":"2024-04-20T02:53:43.334730+0000","last_became_active":"2024-04-20T02:52:54.156070+0000","last_became_peered":"2024-04-20T02:52:54.156070+0000","last_unstale":"2024-04-20T02:53:43.334730+0000","last_undegraded":"2024-04-20T02:53:43.334730+0000","last_fullsized":"2024-04-20T02:53:43.334730+0000","mapping_epoch":154,"log_start":"153'40","ondisk_log_start":"153'40","created":22,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'36","last_scrub_stamp":"2024-04-20T02:52:46.138400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:34.220369+0000","last_clean_scrub_stamp":"2024-04-20T02:52:46.138400+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":40,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:09.709596+0000","scrub_duration":0.003388247,"objects_trimmed":0,"snaptrim_duration":0.038600218999999998,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":116380,"num_write":27,"num_write_kb":7507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1714357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":120989167,"num_objects":48,"num_object_clones":7,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":3077,"num_read_kb":3793692,"num_write":1071,"num_write_kb":295960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":217268250,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":17,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295653376,"data_stored":319204417,"data_compressed":515252,"data_compressed_allocated":434176,"data_compressed_original":1310720,"omap_allocated":2595,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":48,"acting":48,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":66983468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":225505280,"data_stored":240729994,"data_compressed":12805896,"data_compressed_allocated":13840384,"data_compressed_original":29753344,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509487,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76944,"kb_used_data":41732,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94294896,"statfs":{"total":96636764160,"available":96557973504,"internally_reserved":0,"allocated":42733568,"data_stored":45684044,"data_compressed":90764,"data_compressed_allocated":102400,"data_compressed_original":774144,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.44}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.498},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.64400000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.53000000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.435,"15min":0.443},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65500000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.59399999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.441,"15min":0.45000000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57699999999999996,"5min":2.431,"15min":2.431},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.66900000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757283,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103468,"kb_used_data":67544,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94268372,"statfs":{"total":96636764160,"available":96530812928,"internally_reserved":0,"allocated":69165056,"data_stored":72513551,"data_compressed":1987361,"data_compressed_allocated":2129920,"data_compressed_original":4653056,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56599999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42299999999999999,"15min":0.41699999999999998},"min":{"1min":0.34699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.51500000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.47899999999999998,"15min":0.46100000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.437,"15min":0.434},"min":{"1min":0.34499999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.438}]},{"osd":4,"last update":"Sat Apr 20 02:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.29699999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.27900000000000003,"5min":0.252,"15min":0.252},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.33700000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.441,"15min":0.437},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.312},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.49399999999999999,"15min":0.48699999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.36899999999999999}]}]},{"osd":2,"up_from":0,"seq":592705486863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116788,"kb_used_data":81576,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94255052,"statfs":{"total":96636764160,"available":96517173248,"internally_reserved":0,"allocated":83533824,"data_stored":87015843,"data_compressed":1830696,"data_compressed_allocated":1974272,"data_compressed_original":4153344,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.30599999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.26100000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.37},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.41799999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:52:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.753,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.60299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.53000000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.63600000000000001,"5min":1.097,"15min":1.097},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.53500000000000003,"15min":0.54900000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.56399999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.60199999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.51400000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.56100000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574893,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104640,"kb_used_data":69424,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94267200,"statfs":{"total":96636764160,"available":96529612800,"internally_reserved":0,"allocated":71090176,"data_stored":73650740,"data_compressed":1956129,"data_compressed_allocated":2117632,"data_compressed_original":4587520,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.221},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.373}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34799999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.32000000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.85899999999999999,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.52900000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.01,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.44900000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47199999999999998,"15min":0.47399999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.69899999999999995,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54100000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.496},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.55800000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126716,"kb_used_data":91504,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94245124,"statfs":{"total":96636764160,"available":96507006976,"internally_reserved":0,"allocated":93700096,"data_stored":97399851,"data_compressed":2785685,"data_compressed_allocated":3022848,"data_compressed_original":7024640,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.499},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.46700000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:52:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.502,"15min":0.52000000000000002},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.625,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.26800000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.40300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.441,"15min":0.44700000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.57799999999999996,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.5,"15min":0.50600000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.66100000000000003,"5min":1.038,"15min":1.038},"last":0.379}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.434}]}]},{"osd":0,"up_from":15,"seq":64424509483,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119556,"kb_used_data":84336,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94252284,"statfs":{"total":96636764160,"available":96514338816,"internally_reserved":0,"allocated":86360064,"data_stored":89725822,"data_compressed":2819490,"data_compressed_allocated":3063808,"data_compressed_original":6762496,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.435},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.56299999999999994}]},{"osd":4,"last update":"Sat Apr 20 02:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90500000000000003,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69599999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.53000000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.55500000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.58499999999999996}]}]},{"osd":4,"up_from":0,"seq":717259538441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49860608,"data_stored":52937332,"data_compressed":2759802,"data_compressed_allocated":2973696,"data_compressed_original":6184960,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31961088,"data_stored":33949988,"data_compressed":1733638,"data_compressed_allocated":1871872,"data_compressed_original":3956736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12894208,"data_stored":13909415,"data_compressed":734140,"data_compressed_allocated":790528,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18509824,"data_stored":19818665,"data_compressed":1037033,"data_compressed_allocated":1118208,"data_compressed_original":2482176,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34250752,"data_stored":35250075,"data_compressed":54370,"data_compressed_allocated":57344,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49352704,"data_stored":52045922,"data_compressed":97058,"data_compressed_allocated":102400,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2595,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36118528,"data_stored":37574963,"data_compressed":11526,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56590336,"data_stored":67896718,"data_compressed":95804,"data_compressed_allocated":53248,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":34230272,"data_stored":36453414,"data_compressed":42758,"data_compressed_allocated":24576,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":44625920,"data_stored":45837696,"data_compressed":128290,"data_compressed_allocated":114688,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40484864,"data_stored":44145629,"data_compressed":85446,"data_compressed_allocated":69632,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:53:48.819 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-20T02:53:48.819 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-20T02:53:48.833 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:48.833+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:48.871 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:48.871 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:981 4.7 4:e9876a63:::smithi10548823-30:head [write 510946~632952] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 182.28374052999999, 2024-04-20T02:53:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066407731999999997, 2024-04-20T02:53:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.872 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 981 2024-04-20T02:53:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.873 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.874 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585140+0000", 2024-04-20T02:53:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585138+0000", 2024-04-20T02:53:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.876 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585307+0000", 2024-04-20T02:53:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168722 2024-04-20T02:53:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.877 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585333+0000", 2024-04-20T02:53:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6543999999999999e-05 2024-04-20T02:53:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.878 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585348+0000", 2024-04-20T02:53:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5194999999999999e-05 2024-04-20T02:53:48.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.880 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585407+0000", 2024-04-20T02:53:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9070000000000002e-05 2024-04-20T02:53:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.881 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585517+0000", 2024-04-20T02:53:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109344 2024-04-20T02:53:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:48.882 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.585691+0000", 2024-04-20T02:53:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017467700000000001 2024-04-20T02:53:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635237+0000", 2024-04-20T02:53:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049545448999999998 2024-04-20T02:53:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635307+0000", 2024-04-20T02:53:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9752000000000006e-05 2024-04-20T02:53:48.884 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651454+0000", 2024-04-20T02:53:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.01614734 2024-04-20T02:53:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.885 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651489+0000", 2024-04-20T02:53:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5515000000000001e-05 2024-04-20T02:53:48.886 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651548+0000", 2024-04-20T02:53:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8690000000000002e-05 2024-04-20T02:53:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.887 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:986 4.f 4:f72cd604:::smithi10548823-29:head [write 527753~708201,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:48.888 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 182.26301121899999, 2024-04-20T02:53:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070917802000000002, 2024-04-20T02:53:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.890 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 986 2024-04-20T02:53:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:48.891 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605870+0000", 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.605869+0000", 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.892 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606199+0000", 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033054100000000001 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606211+0000", 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1786e-05 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.606220+0000", 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.721e-06 2024-04-20T02:53:48.893 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635917+0000", 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.029697746000000001 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.635968+0000", 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0498999999999999e-05 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.636084+0000", 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000116419 2024-04-20T02:53:48.894 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.667001+0000", 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030916430000000002 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668694+0000", 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0016931229999999999 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.895 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676612+0000", 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0079179379999999994 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676743+0000", 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013081 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.676788+0000", 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4888999999999997e-05 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.896 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:987 4.f 4:f72cd604:::smithi10548823-29:head [write 1856548~737690] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 182.253759468, 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063303489000000004, 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.897 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 987 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615121+0000", 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.898 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615120+0000", 2024-04-20T02:53:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615482+0000", 2024-04-20T02:53:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036256299999999999 2024-04-20T02:53:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.899 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615512+0000", 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9305000000000001e-05 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.615540+0000", 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.832e-05 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.900 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651382+0000", 2024-04-20T02:53:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035842029999999997 2024-04-20T02:53:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.901 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651453+0000", 2024-04-20T02:53:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0658e-05 2024-04-20T02:53:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,4", 2024-04-20T02:53:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.651582+0000", 2024-04-20T02:53:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012911800000000001 2024-04-20T02:53:48.902 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668842+0000", 2024-04-20T02:53:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017260358 2024-04-20T02:53:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.668846+0000", 2024-04-20T02:53:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3930000000000001e-06 2024-04-20T02:53:48.903 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678327+0000", 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0094803430000000005 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678375+0000", 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7930999999999999e-05 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.904 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.678425+0000", 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0265000000000001e-05 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1222 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'70, mlcod=37'65)", 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 174.45676025200001, 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076616765000000003, 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.905 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412121+0000", 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.906 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.412119+0000", 2024-04-20T02:53:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.413996+0000", 2024-04-20T02:53:48.907 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001876249 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414011+0000", 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5429000000000001e-05 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.414022+0000", 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0577000000000001e-05 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.908 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424194+0000", 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010172684 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.424220+0000", 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.5281000000000001e-05 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488698+0000", 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064478124999999997 2024-04-20T02:53:48.909 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488737+0000", 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9660000000000003e-05 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(osd.4.0:269 4.9 e38/22 4:964390c0:::smithi10548823-36:head v 38'71, mlcod=37'65)", 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:48.910 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 174.450755464, 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070631714999999998, 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418125+0000", 2024-04-20T02:53:48.911 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.418124+0000", 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431109+0000", 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012985243 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.912 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431126+0000", 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6945000000000001e-05 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.431137+0000", 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1032e-05 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434348+0000", 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003211256 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.913 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434367+0000", 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.8193e-05 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488744+0000", 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.054377598999999999 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.488757+0000", 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3030000000000001e-05 2024-04-20T02:53:48.914 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4029 4.13 4:cf11ccc0:::smithi10548823-4:head [write 699257~564637,stat] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.313212948, 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.86467550400000004, 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:48.915 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4029 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.916 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555668+0000", 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555666+0000", 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555809+0000", 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014306900000000001 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.917 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555832+0000", 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3680999999999999e-05 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555846+0000", 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3654e-05 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.555913+0000", 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6736999999999995e-05 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.918 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556058+0000", 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014538100000000001 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.556392+0000", 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00033393099999999998 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578661+0000", 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022268553999999999 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.919 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578702+0000", 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1533999999999999e-05 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578747+0000", 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4858999999999999e-05 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578811+0000", 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3520000000000002e-05 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.920 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605743+0000", 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026932418999999999 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420312+0000", 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81456849600000003 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420343+0000", 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1911e-05 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.921 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4030 4.13 4:cf11ccc0:::smithi10548823-4:head [write 1873062~640438] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.306299826999997, 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85780235299999996, 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4030 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562581+0000", 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.923 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562579+0000", 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562714+0000", 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013475 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562728+0000", 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3516000000000001e-05 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.924 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.562736+0000", 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5690000000000005e-06 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578835+0000", 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.016099156 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.578957+0000", 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121094 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.925 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.579123+0000", 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016647100000000001 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605727+0000", 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.026603643999999999 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605749+0000", 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2070999999999999e-05 2024-04-20T02:53:48.926 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605799+0000", 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0417000000000002e-05 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605841+0000", 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2344e-05 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605907+0000", 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5135999999999994e-05 2024-04-20T02:53:48.927 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420373+0000", 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81446675899999998 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420383+0000", 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0108999999999999e-05 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.928 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4031 4.13 4:cf11ccc0:::smithi10548823-4:head [write 3305941~593316] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.298738252, 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85027709799999995, 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4031 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.929 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570143+0000", 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570142+0000", 2024-04-20T02:53:48.930 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570277+0000", 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000135606 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570288+0000", 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0719e-05 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.931 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570294+0000", 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2519999999999996e-06 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592786+0000", 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.022491220999999999 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.592911+0000", 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012546599999999999 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.932 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.593118+0000", 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000207011 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605878+0000", 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.012760298999999999 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605934+0000", 2024-04-20T02:53:48.933 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5813999999999998e-05 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605963+0000", 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.9275e-05 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606612+0000", 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00064813999999999996 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.934 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613782+0000", 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0071705670000000001 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420411+0000", 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.80662907399999995 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420420+0000", 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5790000000000004e-06 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.935 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4032 4.13 4:cf11ccc0:::smithi10548823-4:head [setxattr _header (56) in=7b,truncate 3899257] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.298581824000003, 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85015831100000006, 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4032 2024-04-20T02:53:48.936 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570299+0000", 2024-04-20T02:53:48.937 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570302+0000", 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.1700000000000001e-06 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570305+0000", 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.976e-06 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.938 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570308+0000", 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8899999999999999e-06 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.605985+0000", 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035677121999999999 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606066+0000", 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0797000000000006e-05 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.939 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,4,5,6,7", 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606211+0000", 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014569999999999999 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607013+0000", 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00080160599999999995 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607046+0000", 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.2938999999999998e-05 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.940 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607086+0000", 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0203000000000001e-05 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.607157+0000", 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0621e-05 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.613833+0000", 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0066762269999999999 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.941 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420448+0000", 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.806615478 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420457+0000", 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9779999999999994e-06 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.942 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4033 4.13 4:cf11ccc0:::smithi10548823-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.298569950000001, 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85021192700000003, 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4033 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.943 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:48.945 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570311+0000", 2024-04-20T02:53:48.946 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570312+0000", 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6419999999999999e-06 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570316+0000", 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3730000000000001e-06 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570318+0000", 2024-04-20T02:53:48.947 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6680000000000001e-06 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606541+0000", 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036222797000000001 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606598+0000", 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6629999999999998e-05 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420515+0000", 2024-04-20T02:53:48.948 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81391740099999998 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420523+0000", 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.6799999999999993e-06 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4034 4.13 4:cfd1e4d4:::smithi10548823-9:head [rollback 3f] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 60.298560311999999, 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.85017147900000001, 2024-04-20T02:53:48.949 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "reached pg", 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4034 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.950 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570321+0000", 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570320+0000", 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967296 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570322+0000", 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.37e-06 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.951 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570325+0000", 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7800000000000001e-06 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.570328+0000", 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5379999999999998e-06 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606632+0000", 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036304075999999998 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.952 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:48.606716+0000", 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3689000000000005e-05 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420484+0000", 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.81376812700000001 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:49.420492+0000", 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1179999999999992e-06 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.953 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4062 4.9 4:9c87d36d:::smithi10548823-24:head [write 766485~465633] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.718706531000002, 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075029242999999995, 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4062 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.954 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150174+0000", 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.955 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150173+0000", 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150435+0000", 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000262064 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150445+0000", 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8160000000000005e-06 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.956 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150450+0000", 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1880000000000001e-06 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150484+0000", 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150490+0000", 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6810000000000003e-06 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.957 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161686+0000", 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011196187 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.161840+0000", 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015389000000000001 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.162103+0000", 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026333299999999998 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.958 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224927+0000", 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062823458999999998 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.224993+0000", 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6667000000000004e-05 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225059+0000", 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6117000000000004e-05 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.959 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225119+0000", 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9342000000000002e-05 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225204+0000", 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4844000000000003e-05 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4063 4.9 4:9c87d36d:::smithi10548823-24:head [write 1990637~647776] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.960 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.718423856999998, 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075097435000000004, 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4063 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.961 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150457+0000", 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.962 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150746+0000", 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00028951399999999999 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150753+0000", 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0999999999999998e-06 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150757+0000", 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2300000000000002e-06 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.963 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150801+0000", 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3421000000000003e-05 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150811+0000", 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0581000000000001e-05 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177065+0000", 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.02625367 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.964 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177164+0000", 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.8900999999999994e-05 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177319+0000", 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000155437 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225048+0000", 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.047728857 2024-04-20T02:53:48.965 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225217+0000", 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000168395 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225375+0000", 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000157837 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225437+0000", 2024-04-20T02:53:48.966 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2824999999999996e-05 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225555+0000", 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117169 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4064 4.9 4:9c87d36d:::smithi10548823-24:head [write 3299280~667205] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.718117104000001, 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112222372, 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.967 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4064 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150764+0000", 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.968 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.150763+0000", 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151076+0000", 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000312355 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151083+0000", 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1010000000000003e-06 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.969 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151091+0000", 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0409999999999998e-06 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151127+0000", 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.608e-05 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151133+0000", 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1099999999999999e-06 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.970 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200197+0000", 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049064000000000003 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200315+0000", 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117855 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.200545+0000", 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022990700000000001 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.971 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225595+0000", 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.025050224999999999 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262670+0000", 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075460999999997 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262767+0000", 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6479999999999998e-05 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262835+0000", 2024-04-20T02:53:48.972 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7999999999999999e-05 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262986+0000", 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015129799999999999 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4065 4.9 4:9c87d36d:::smithi10548823-24:head [setxattr _header (57) in=7b,truncate 3966485] snapc 47=[47,45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.717777224000002, 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112193353, 2024-04-20T02:53:48.973 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4065 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.974 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151104+0000", 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151103+0000", 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151107+0000", 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8870000000000002e-06 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.975 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151111+0000", 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2309999999999998e-06 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151116+0000", 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4850000000000003e-06 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151168+0000", 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2673000000000002e-05 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.976 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151179+0000", 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0112000000000001e-05 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225611+0000", 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074432549000000001 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225689+0000", 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.7612000000000002e-05 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.977 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.225858+0000", 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00016945400000000001 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262650+0000", 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036792126000000001 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262678+0000", 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.8132000000000001e-05 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.978 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263077+0000", 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00039868299999999999 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263141+0000", 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3448999999999995e-05 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263297+0000", 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015649699999999999 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.979 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4066 4.9 4:9c87d36d:::smithi10548823-24:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.717760368, 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112663448, 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4066 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.980 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151121+0000", 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151120+0000", 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.981 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151123+0000", 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.4389999999999999e-06 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151128+0000", 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2449999999999998e-06 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151132+0000", 2024-04-20T02:53:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9049999999999999e-06 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151198+0000", 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5988000000000006e-05 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151205+0000", 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8770000000000004e-06 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226171+0000", 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074966844000000005 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226261+0000", 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9975999999999994e-05 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263337+0000", 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037075863000000001 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.984 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263443+0000", 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000105678 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263784+0000", 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034103200000000001 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.985 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4095 4.9 4:964390c0:::smithi10548823-36:head [write 450520~558231] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.717746298000002, 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112875926, 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4095 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:48.986 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151135+0000", 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151134+0000", 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151379+0000", 2024-04-20T02:53:48.987 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024486000000000001 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151385+0000", 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1460000000000001e-06 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151389+0000", 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6610000000000002e-06 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151413+0000", 2024-04-20T02:53:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3611000000000001e-05 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151419+0000", 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8780000000000003e-06 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226302+0000", 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.074883540999999998 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226460+0000", 2024-04-20T02:53:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015792000000000001 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.226599+0000", 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013939599999999999 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262685+0000", 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036086115000000002 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.262726+0000", 2024-04-20T02:53:48.990 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0448999999999998e-05 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263833+0000", 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001106685 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263884+0000", 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1530999999999998e-05 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264011+0000", 2024-04-20T02:53:48.991 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012641900000000001 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4096 4.9 4:964390c0:::smithi10548823-36:head [write 1483047~515125] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.717486762999997, 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11703358799999999, 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.992 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4096 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:48.993 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151394+0000", 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151644+0000", 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025029799999999998 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151651+0000", 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.148e-06 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:48.994 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151655+0000", 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2769999999999999e-06 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151689+0000", 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.3711999999999998e-05 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151698+0000", 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0359999999999995e-06 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.995 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245111+0000", 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.093413164000000007 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245232+0000", 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000120214 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.245424+0000", 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019272500000000001 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.996 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.263033+0000", 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.017608461999999998 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266386+0000", 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0033530980000000001 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268218+0000", 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001832101 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.997 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268283+0000", 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5334999999999997e-05 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268428+0000", 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014449999999999999 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.998 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4097 4.9 4:964390c0:::smithi10548823-36:head [setxattr _header (57) in=7b,truncate 2050520] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.717218809999999, 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.117031099, 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4097 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:48.999 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151662+0000", 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.000 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151665+0000", 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.614e-06 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151670+0000", 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1340000000000003e-06 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151675+0000", 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0560000000000004e-06 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.001 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151738+0000", 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.2447999999999996e-05 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151753+0000", 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.5007e-05 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264044+0000", 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.112291562 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.002 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264165+0000", 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000121081 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 3,6", 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264421+0000", 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025568599999999998 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266133+0000", 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001711668 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.003 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.266755+0000", 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00062224100000000005 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268484+0000", 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001729169 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268545+0000", 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0365000000000002e-05 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.004 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268693+0000", 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014860000000000001 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4098 4.9 4:964390c0:::smithi10548823-36:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 56.717201328000002, 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11735205, 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.005 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4098 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151680+0000", 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151679+0000", 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151681+0000", 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.3140000000000002e-06 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.007 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151686+0000", 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0209999999999999e-06 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151691+0000", 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4700000000000004e-06 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151779+0000", 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.8399999999999994e-05 2024-04-20T02:53:49.008 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.151788+0000", 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.7129999999999994e-06 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265294+0000", 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.11350631899999999 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.265391+0000", 2024-04-20T02:53:49.009 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6861000000000007e-05 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268730+0000", 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003339102 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.268841+0000", 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110708 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.010 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.269032+0000", 2024-04-20T02:53:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019057899999999999 2024-04-20T02:53:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.011 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.011 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:49.012 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-20T02:53:49.013 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:49.013+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:49.113 INFO:tasks.ceph.osd.5.smithi138.stderr:2024-04-20T02:53:49.114+0000 7fcb9603a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 52585) UID: 0 2024-04-20T02:53:49.142 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:49.142 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:49.143 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:49.143 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:49.154 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-20T02:53:49.214 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:49.214+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:49.282 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:49.282 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:49.282 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:49.292 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-20T02:53:49.314 INFO:tasks.ceph.osd.7.smithi138.stderr:2024-04-20T02:53:49.315+0000 7fdd3a741700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 43325) UID: 0 2024-04-20T02:53:49.378 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'unique_pool_0' removed 2024-04-20T02:53:49.383 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is set 2024-04-20T02:53:49.397 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-20T02:53:49.408 INFO:tasks.thrashosds:joining thrashosds 2024-04-20T02:53:49.435 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [ 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:968 4.0 4:007b2a87:::smithi10548823-39:head [write 477332~766101,stat] snapc 2=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 182.890908131, 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.063642761000000006, 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 968 2024-04-20T02:53:49.436 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543740+0000", 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.543738+0000", 2024-04-20T02:53:49.437 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544101+0000", 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00036369199999999999 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544121+0000", 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.9763999999999999e-05 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.438 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544136+0000", 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4645e-05 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544225+0000", 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.9829999999999999e-05 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544340+0000", 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00011404 2024-04-20T02:53:49.439 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.544535+0000", 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019571200000000001 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.594581+0000", 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.050046039 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.440 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.595313+0000", 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00073154700000000003 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607290+0000", 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011976977 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.441 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607338+0000", 2024-04-20T02:53:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8653999999999999e-05 2024-04-20T02:53:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:46.607382+0000", 2024-04-20T02:53:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:49.442 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:1210 4.0 4:0d9b1ec7:::smithi10548823-14:head [write 1593482~658027] snapc 9=[9,6] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 175.062359243, 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.062133961000000001, 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.443 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 1210 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372288+0000", 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.444 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372287+0000", 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372627+0000", 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034049999999999998 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372642+0000", 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4676e-05 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.445 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.372653+0000", 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0804e-05 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386661+0000", 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.014008227999999999 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386776+0000", 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000114906 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.446 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.386973+0000", 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019724 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423293+0000", 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036320230000000002 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.423469+0000", 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017532800000000001 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.447 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434312+0000", 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010843521 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434371+0000", 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9138000000000002e-05 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.434422+0000", 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.117e-05 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.448 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:1221 4.9 e38/22 4:95e19a88:::smithi10548823-40:head v 38'69, mlcod=37'65)", 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 175.04610706099999, 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070088839, 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.449 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388541+0000", 2024-04-20T02:53:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.450 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.388540+0000", 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390275+0000", 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001735257 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.451 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390287+0000", 2024-04-20T02:53:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1267000000000001e-05 2024-04-20T02:53:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390295+0000", 2024-04-20T02:53:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.2439999999999993e-06 2024-04-20T02:53:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.452 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390356+0000", 2024-04-20T02:53:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1321999999999993e-05 2024-04-20T02:53:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.390377+0000", 2024-04-20T02:53:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.067e-05 2024-04-20T02:53:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.453 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458579+0000", 2024-04-20T02:53:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.068202468000000002 2024-04-20T02:53:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.454 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:50:54.458629+0000", 2024-04-20T02:53:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9969000000000002e-05 2024-04-20T02:53:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:2699 4.0 4:007b2a87:::smithi10548823-39:head [write 648803~722611] snapc 2c=[2c] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-04-20T02:53:49.455 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 117.086408042, 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070173018000000004, 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 2699 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:49.456 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348240+0000", 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348237+0000", 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348621+0000", 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038390199999999998 2024-04-20T02:53:49.457 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348643+0000", 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2234000000000001e-05 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348693+0000", 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.9035000000000001e-05 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348774+0000", 2024-04-20T02:53:49.458 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1604e-05 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.348860+0000", 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5987000000000002e-05 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,7", 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.349012+0000", 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015236899999999999 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390452+0000", 2024-04-20T02:53:49.459 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.041439225000000003 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.390461+0000", 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.1889999999999996e-06 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418322+0000", 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.027861123000000002 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418370+0000", 2024-04-20T02:53:49.460 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7861000000000001e-05 2024-04-20T02:53:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:51:52.418413+0000", 2024-04-20T02:53:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2731000000000001e-05 2024-04-20T02:53:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.461 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3872 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 779924~785847,stat] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.084004802999999, 2024-04-20T02:53:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070878165000000007, 2024-04-20T02:53:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.462 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3872 2024-04-20T02:53:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.463 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350643+0000", 2024-04-20T02:53:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.350641+0000", 2024-04-20T02:53:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:49.464 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351179+0000", 2024-04-20T02:53:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00053797199999999995 2024-04-20T02:53:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351196+0000", 2024-04-20T02:53:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.66e-05 2024-04-20T02:53:49.465 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351209+0000", 2024-04-20T02:53:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2952000000000001e-05 2024-04-20T02:53:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.466 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351306+0000", 2024-04-20T02:53:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.7272e-05 2024-04-20T02:53:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351322+0000", 2024-04-20T02:53:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.6416e-05 2024-04-20T02:53:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.467 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358164+0000", 2024-04-20T02:53:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0068416019999999996 2024-04-20T02:53:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358336+0000", 2024-04-20T02:53:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017156499999999999 2024-04-20T02:53:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.468 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.358594+0000", 2024-04-20T02:53:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025794499999999998 2024-04-20T02:53:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389148+0000", 2024-04-20T02:53:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.030554773 2024-04-20T02:53:49.469 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409688+0000", 2024-04-20T02:53:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.020539497 2024-04-20T02:53:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420893+0000", 2024-04-20T02:53:49.470 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011205503 2024-04-20T02:53:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420934+0000", 2024-04-20T02:53:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1171000000000003e-05 2024-04-20T02:53:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.471 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420998+0000", 2024-04-20T02:53:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3386999999999995e-05 2024-04-20T02:53:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421521+0000", 2024-04-20T02:53:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00052313200000000005 2024-04-20T02:53:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.472 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3873 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 2078257~580112] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.083421517000005, 2024-04-20T02:53:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075737411000000004, 2024-04-20T02:53:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.473 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3873 2024-04-20T02:53:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.474 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351226+0000", 2024-04-20T02:53:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.475 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351225+0000", 2024-04-20T02:53:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351614+0000", 2024-04-20T02:53:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00038825099999999999 2024-04-20T02:53:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.476 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351625+0000", 2024-04-20T02:53:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1173e-05 2024-04-20T02:53:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351634+0000", 2024-04-20T02:53:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0489999999999993e-06 2024-04-20T02:53:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.477 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351696+0000", 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1885999999999997e-05 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351708+0000", 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2702e-05 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389193+0000", 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037484574999999999 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.478 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389303+0000", 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000110188 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.389563+0000", 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00026034499999999999 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.420846+0000", 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.031283014999999997 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.479 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421592+0000", 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00074529299999999995 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421632+0000", 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0677999999999999e-05 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426683+0000", 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0050507119999999997 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.480 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426752+0000", 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.8484000000000004e-05 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426964+0000", 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021193500000000001 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3874 4.10 4:0d9b1ec7:::smithi10548823-14:head [write 3301257~278667] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:49.481 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.082986516000005, 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075720365999999997, 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3874 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:49.482 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351661+0000", 2024-04-20T02:53:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.483 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351660+0000", 2024-04-20T02:53:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.484 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351852+0000", 2024-04-20T02:53:49.485 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00019116900000000001 2024-04-20T02:53:49.485 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.485 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.485 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.485 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351861+0000", 2024-04-20T02:53:49.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.6050000000000004e-06 2024-04-20T02:53:49.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.486 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351869+0000", 2024-04-20T02:53:49.486 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.4780000000000002e-06 2024-04-20T02:53:49.486 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.486 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.486 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351916+0000", 2024-04-20T02:53:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.7222999999999998e-05 2024-04-20T02:53:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351927+0000", 2024-04-20T02:53:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0791000000000001e-05 2024-04-20T02:53:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409721+0000", 2024-04-20T02:53:49.488 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.057793887000000002 2024-04-20T02:53:49.488 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.488 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.488 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.488 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.409840+0000", 2024-04-20T02:53:49.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000119837 2024-04-20T02:53:49.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.489 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:49.489 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.410063+0000", 2024-04-20T02:53:49.489 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022257200000000001 2024-04-20T02:53:49.489 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.489 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421678+0000", 2024-04-20T02:53:49.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.011615448 2024-04-20T02:53:49.490 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.490 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.490 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.490 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421721+0000", 2024-04-20T02:53:49.490 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.2191000000000003e-05 2024-04-20T02:53:49.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.491 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.423989+0000", 2024-04-20T02:53:49.491 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0022682879999999998 2024-04-20T02:53:49.491 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.491 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.491 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427051+0000", 2024-04-20T02:53:49.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003061785 2024-04-20T02:53:49.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.492 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.492 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427157+0000", 2024-04-20T02:53:49.492 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000106684 2024-04-20T02:53:49.492 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.492 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.493 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.493 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427381+0000", 2024-04-20T02:53:49.493 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022416299999999999 2024-04-20T02:53:49.493 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.493 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.493 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.493 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.493 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.494 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3875 4.10 4:0d9b1ec7:::smithi10548823-14:head [setxattr _header (57) in=7b,truncate 3579924] snapc 45=[45,41,3f,39] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:49.494 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:49.494 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.082765355000006, 2024-04-20T02:53:49.494 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.075851731000000006, 2024-04-20T02:53:49.494 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.494 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.494 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.494 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.494 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3875 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.495 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351882+0000", 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351886+0000", 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.561e-06 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351893+0000", 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.6100000000000002e-06 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.496 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351899+0000", 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8660000000000002e-06 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352002+0000", 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103811 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352012+0000", 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4110000000000002e-06 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.497 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421749+0000", 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.069736690000000004 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.421861+0000", 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112587 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422120+0000", 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00025878099999999997 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.498 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424547+0000", 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0024270379999999999 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.424960+0000", 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00041259899999999998 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.426257+0000", 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001297231 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.499 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427489+0000", 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0012320949999999999 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427562+0000", 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.2825999999999997e-05 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427734+0000", 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00017230699999999999 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.500 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:3876 4.10 4:0d9b1ec7:::smithi10548823-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 65.082743449000006, 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.076230629999999994, 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 3876 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.501 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351904+0000", 2024-04-20T02:53:49.502 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.503 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.503 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351906+0000", 2024-04-20T02:53:49.503 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.6350000000000002e-06 2024-04-20T02:53:49.503 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.503 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351912+0000", 2024-04-20T02:53:49.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8209999999999998e-06 2024-04-20T02:53:49.504 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.504 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.504 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.504 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.351918+0000", 2024-04-20T02:53:49.504 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3839999999999996e-06 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352035+0000", 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000117207 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.352043+0000", 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.0169999999999996e-06 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.505 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422511+0000", 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.070467815000000003 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.422619+0000", 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010819600000000001 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.506 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427772+0000", 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.005153398 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.427900+0000", 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012740399999999999 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:44.428135+0000", 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235138 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.507 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4067 4.c 4:392c1e2f:::smithi10548823-44:head [write 783292~540882] snapc 47=[47,45,3f] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.292051239999999, 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072910900000000001, 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4067 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.508 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142596+0000", 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142595+0000", 2024-04-20T02:53:49.509 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142941+0000", 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034585799999999999 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142954+0000", 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3491e-05 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.510 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142965+0000", 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1139e-05 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143003+0000", 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7499000000000001e-05 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143011+0000", 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.5490000000000006e-06 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156033+0000", 2024-04-20T02:53:49.511 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.013021576999999999 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156242+0000", 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00020885500000000001 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.156474+0000", 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023195900000000001 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.177835+0000", 2024-04-20T02:53:49.512 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.021361594000000001 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178072+0000", 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000237304 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.214963+0000", 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.036890158999999999 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.513 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215046+0000", 2024-04-20T02:53:49.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3399000000000006e-05 2024-04-20T02:53:49.514 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.514 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.514 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.514 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215507+0000", 2024-04-20T02:53:49.514 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00046127700000000001 2024-04-20T02:53:49.514 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.514 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.514 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.515 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.515 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.515 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4068 4.c 4:392c1e2f:::smithi10548823-44:head [write 1932766~507787] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:49.515 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:49.515 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.291672296999998, 2024-04-20T02:53:49.515 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073156952999999997, 2024-04-20T02:53:49.515 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.515 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.515 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.515 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4068 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.516 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.142975+0000", 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143300+0000", 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000325225 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143308+0000", 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.3860000000000007e-06 2024-04-20T02:53:49.517 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143315+0000", 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7000000000000002e-06 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143360+0000", 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5475000000000001e-05 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143373+0000", 2024-04-20T02:53:49.518 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2204e-05 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178109+0000", 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034736547999999999 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178222+0000", 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000112959 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.178452+0000", 2024-04-20T02:53:49.519 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000229686 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215864+0000", 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.037412038000000002 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215878+0000", 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.4029000000000001e-05 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215929+0000", 2024-04-20T02:53:49.520 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0738000000000001e-05 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.215989+0000", 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.0350999999999999e-05 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216132+0000", 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014332099999999999 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.521 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4069 4.c 4:392c1e2f:::smithi10548823-44:head [write 3210350~572013] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.291323484000003, 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108473219, 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.522 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4069 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.523 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.524 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.524 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143324+0000", 2024-04-20T02:53:49.524 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.524 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.524 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.525 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143664+0000", 2024-04-20T02:53:49.525 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00034090500000000002 2024-04-20T02:53:49.525 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.525 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.525 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143673+0000", 2024-04-20T02:53:49.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.1119999999999996e-06 2024-04-20T02:53:49.526 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.526 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.526 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.526 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143679+0000", 2024-04-20T02:53:49.526 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.3589999999999996e-06 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143724+0000", 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.4974e-05 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143734+0000", 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.0275000000000001e-05 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.527 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197473+0000", 2024-04-20T02:53:49.528 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053738971000000003 2024-04-20T02:53:49.528 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.528 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.528 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.528 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197616+0000", 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00014275900000000001 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.197834+0000", 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021806000000000001 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251517+0000", 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.053683445000000003 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.529 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251579+0000", 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.1255999999999998e-05 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251590+0000", 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1399e-05 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251657+0000", 2024-04-20T02:53:49.530 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.7039999999999995e-05 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251797+0000", 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000140235 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4070 4.c 4:392c1e2f:::smithi10548823-44:head [setxattr _header (57) in=7b,truncate 3983292] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.290960695999999, 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108344988, 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.531 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4070 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143687+0000", 2024-04-20T02:53:49.532 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143686+0000", 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143690+0000", 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7579999999999998e-06 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143697+0000", 2024-04-20T02:53:49.533 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.0119999999999999e-06 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143702+0000", 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.78e-06 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143758+0000", 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.5640999999999998e-05 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.534 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143766+0000", 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.9519999999999994e-06 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216172+0000", 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.072406561999999994 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216295+0000", 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000122452 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.535 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216531+0000", 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00023635999999999999 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251815+0000", 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035283760999999997 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251856+0000", 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.1270999999999998e-05 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.536 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251892+0000", 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.5663000000000001e-05 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251929+0000", 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.7055999999999999e-05 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252032+0000", 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103183 2024-04-20T02:53:49.537 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4071 4.c 4:392c1e2f:::smithi10548823-44:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.290925497000003, 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108670215, 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:49.538 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4071 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143722+0000", 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.539 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143721+0000", 2024-04-20T02:53:49.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:49.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.540 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.540 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143727+0000", 2024-04-20T02:53:49.540 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.2510000000000002e-06 2024-04-20T02:53:49.540 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.540 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143739+0000", 2024-04-20T02:53:49.541 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2649e-05 2024-04-20T02:53:49.541 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.541 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.541 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.541 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143749+0000", 2024-04-20T02:53:49.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.3340000000000008e-06 2024-04-20T02:53:49.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.542 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.542 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.542 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143788+0000", 2024-04-20T02:53:49.542 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9773e-05 2024-04-20T02:53:49.542 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.543 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143796+0000", 2024-04-20T02:53:49.543 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.8720000000000002e-06 2024-04-20T02:53:49.543 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.543 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.543 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216846+0000", 2024-04-20T02:53:49.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.073049823 2024-04-20T02:53:49.544 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.544 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.544 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:49.544 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.216948+0000", 2024-04-20T02:53:49.544 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000101525 2024-04-20T02:53:49.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.545 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.545 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.545 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252070+0000", 2024-04-20T02:53:49.545 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.035122911999999999 2024-04-20T02:53:49.545 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.546 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252180+0000", 2024-04-20T02:53:49.546 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010907499999999999 2024-04-20T02:53:49.546 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.546 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.546 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.547 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252392+0000", 2024-04-20T02:53:49.547 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021285400000000001 2024-04-20T02:53:49.547 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.547 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.547 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.547 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.547 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.548 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4106 4.c 4:3d524298:::smithi10548823-18:head [write 484134~708729] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:49.548 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:49.548 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.290892392000003, 2024-04-20T02:53:49.548 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108955254, 2024-04-20T02:53:49.548 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.548 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4106 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.549 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.143755+0000", 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144188+0000", 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000433771 2024-04-20T02:53:49.550 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144198+0000", 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.4970000000000007e-06 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144205+0000", 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.3409999999999997e-06 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145714+0000", 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0015088110000000001 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.551 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145727+0000", 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.2989e-05 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217011+0000", 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.071283814000000001 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217168+0000", 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00015707300000000001 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.552 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.217397+0000", 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00022881900000000001 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.251870+0000", 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.034473539999999997 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252459+0000", 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000589151 2024-04-20T02:53:49.553 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252511+0000", 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.1282999999999997e-05 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252569+0000", 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.8294999999999997e-05 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252711+0000", 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000141435 2024-04-20T02:53:49.554 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4107 4.c 4:3d524298:::smithi10548823-18:head [write 1883658~200476] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.290430327000003, 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.10880477099999999, 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.555 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4107 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144217+0000", 2024-04-20T02:53:49.557 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144216+0000", 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144346+0000", 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000129732 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.558 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144355+0000", 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 8.4300000000000006e-06 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144361+0000", 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.9150000000000001e-06 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145754+0000", 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0013929649999999999 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145763+0000", 2024-04-20T02:53:49.559 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2990000000000002e-06 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243874+0000", 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.098111462999999996 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.243990+0000", 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0001156 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.244226+0000", 2024-04-20T02:53:49.560 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000235839 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252760+0000", 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0085345439999999998 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252817+0000", 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.6842000000000002e-05 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252829+0000", 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.1817999999999999e-05 2024-04-20T02:53:49.561 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.252884+0000", 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.4610000000000001e-05 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253022+0000", 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013853800000000001 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.562 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4108 4.c 4:3d524298:::smithi10548823-18:head [setxattr _header (57) in=7b,truncate 2084134] snapc 47=[45,3f] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.290281903, 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.120324168, 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4108 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:49.563 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144366+0000", 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144365+0000", 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144369+0000", 2024-04-20T02:53:49.564 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.8500000000000004e-06 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144374+0000", 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.0080000000000002e-06 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144379+0000", 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.8239999999999999e-06 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145790+0000", 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0014115449999999999 2024-04-20T02:53:49.565 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145800+0000", 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.0429999999999996e-06 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253101+0000", 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.107301173 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253210+0000", 2024-04-20T02:53:49.566 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000109146 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 1,6", 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.253456+0000", 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00024599700000000002 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254623+0000", 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.001166718 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.258220+0000", 2024-04-20T02:53:49.567 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.003597135 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264486+0000", 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0062665819999999997 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264558+0000", 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 7.1364000000000003e-05 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264690+0000", 2024-04-20T02:53:49.568 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00013227099999999999 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4109 4.c 4:3d524298:::smithi10548823-18:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.290264381, 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.12051582399999999, 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "started", 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.569 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4109 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.570 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144383+0000", 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144385+0000", 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.2869999999999999e-06 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144392+0000", 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.5749999999999997e-06 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.571 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.144397+0000", 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 5.3589999999999999e-06 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145829+0000", 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00143191 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for peered", 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.145838+0000", 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 9.2390000000000008e-06 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.572 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254052+0000", 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.108214029 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.254156+0000", 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00010343799999999999 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264726+0000", 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.010570144 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.573 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264796+0000", 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 6.9886000000000001e-05 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.264899+0000", 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000103336 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_repop(client.4431.0:4146 3.1 e153/152 3:83fccc85:::55db2dc496ef8be4fec7f179ff147f3304f90df32fa9056392a715418b670b22:head v 153'228, mlcod=0'0)", 2024-04-20T02:53:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 57.252868194999998, 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.066047175, 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181779+0000", 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.575 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181778+0000", 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.999999 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.181997+0000", 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00021896 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182011+0000", 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.3339999999999999e-05 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.576 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182021+0000", 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 1.046e-05 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182068+0000", 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.6931999999999997e-05 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.182104+0000", 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.6321e-05 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.577 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247786+0000", 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.065681787000000005 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:52.247827+0000", 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.0491000000000002e-05 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: "description": "osd_op(client.4431.0:4183 3.2 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head [write 0~2392064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-04-20T02:53:49.578 INFO:teuthology.orchestra.run.smithi105.stdout: "initiated_at": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "age": 55.783234022999999, 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.064859071000000004, 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "type_data": { 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "client_info": { 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "client": "client.4431", 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "tid": 4183 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "events": [ 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "initiated", 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.579 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "throttled", 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651414+0000", 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "header_read", 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.651396+0000", 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4294967295.9999819 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "all_read", 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652285+0000", 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00088949500000000002 2024-04-20T02:53:49.580 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "dispatched", 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652306+0000", 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.0812e-05 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "queued_for_pg", 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652333+0000", 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 2.7086000000000001e-05 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "reached_pg", 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652373+0000", 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9690000000000001e-05 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.581 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "started", 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652499+0000", 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.00012642299999999999 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "waiting for subops from 5,6", 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.652608+0000", 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.000108661 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "op_commit", 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.702325+0000", 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.049717048999999999 2024-04-20T02:53:49.582 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.711665+0000", 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.0093403370000000006 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716188+0000", 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 0.004522997 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "commit_sent", 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716227+0000", 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 3.9240999999999999e-05 2024-04-20T02:53:49.583 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-04-20T02:53:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: { 2024-04-20T02:53:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: "event": "done", 2024-04-20T02:53:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: "time": "2024-04-20T02:52:53.716273+0000", 2024-04-20T02:53:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 4.5228000000000001e-05 2024-04-20T02:53:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-04-20T02:53:49.584 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-04-20T02:53:49.584 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:49.585 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-20T02:53:49.725 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:49.726 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:49.726 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:49.726 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:49.736 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-20T02:53:49.864 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:49.864 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:49.864 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:49.864 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:49.864 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:49.874 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-20T02:53:50.003 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:50.003 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:50.003 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:50.003 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:50.003 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:137 4.a 4:5fc9a13e:::smithi10548823-35:head [write 571438~623878,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 188.26718873999999, 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071983310999999994, 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 137 2024-04-20T02:53:50.004 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735382+0000", 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.005 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.735380+0000", 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739020+0000", 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0036404950000000001 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739031+0000", 2024-04-20T02:53:50.006 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0769e-05 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739038+0000", 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9260000000000003e-06 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739080+0000", 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2021999999999998e-05 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739183+0000", 2024-04-20T02:53:50.007 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010331300000000001 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.739332+0000", 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000148224 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776373+0000", 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037041050999999998 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.008 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776379+0000", 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.161e-06 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807309+0000", 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030929943000000001 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807339+0000", 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9742000000000001e-05 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.009 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807365+0000", 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6296e-05 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:145 4.a 4:5a553643:::smithi10548823-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 188.235899679, 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072057298000000006, 2024-04-20T02:53:50.010 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 145 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.011 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766671+0000", 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.766669+0000", 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767675+0000", 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00100589 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767688+0000", 2024-04-20T02:53:50.012 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3438000000000001e-05 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.767696+0000", 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9300000000000003e-06 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776482+0000", 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.008785892 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776587+0000", 2024-04-20T02:53:50.013 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010435800000000001 2024-04-20T02:53:50.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:50.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.776725+0000", 2024-04-20T02:53:50.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000138658 2024-04-20T02:53:50.014 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.014 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.014 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.014 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.807383+0000", 2024-04-20T02:53:50.014 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030657469999999999 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808323+0000", 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00094051699999999998 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838639+0000", 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030315587000000001 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838686+0000", 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6715999999999998e-05 2024-04-20T02:53:50.015 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838728+0000", 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2415e-05 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:146 4.a 4:5a553643:::smithi10548823-37:head [write 1923587~633336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:50.016 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 188.23120922300001, 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067459599999999995, 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 146 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.017 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771361+0000", 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.771360+0000", 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772223+0000", 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00086259800000000001 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772232+0000", 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8720000000000008e-06 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.772237+0000", 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1070000000000004e-06 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793785+0000", 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02154795 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793842+0000", 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.7133999999999999e-05 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.793928+0000", 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5688999999999999e-05 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835729+0000", 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041801521000000001 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835837+0000", 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010798600000000001 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838757+0000", 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029192419999999998 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838791+0000", 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3933000000000002e-05 2024-04-20T02:53:50.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838821+0000", 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0171e-05 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:149 4.a 4:5fd945b2:::smithi10548823-38:head [write 621859~449625,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 188.224452754, 2024-04-20T02:53:50.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081016687000000004, 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 149 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778118+0000", 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778116+0000", 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778661+0000", 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000545288 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778675+0000", 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3692e-05 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.778684+0000", 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4870000000000002e-06 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808349+0000", 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.029665128999999998 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808438+0000", 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8997000000000002e-05 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.808533+0000", 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5329999999999997e-05 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835830+0000", 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027296731000000001 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835851+0000", 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1081e-05 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859042+0000", 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023191479000000001 2024-04-20T02:53:50.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859096+0000", 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3405000000000001e-05 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.859134+0000", 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8417999999999997e-05 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:150 4.a 4:5fd945b2:::smithi10548823-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 188.214637243, 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080368060000000005, 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 150 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787933+0000", 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.787932+0000", 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788946+0000", 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010144469999999999 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788959+0000", 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3064999999999999e-05 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788970+0000", 2024-04-20T02:53:50.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0339999999999999e-05 2024-04-20T02:53:50.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819522+0000", 2024-04-20T02:53:50.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030552246000000002 2024-04-20T02:53:50.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819583+0000", 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1060000000000002e-05 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.819664+0000", 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0777000000000003e-05 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.838831+0000", 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.019167274000000002 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841467+0000", 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026360730000000001 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868212+0000", 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026745134 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868262+0000", 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.9354000000000002e-05 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868301+0000", 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9449999999999997e-05 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:151 4.a 4:5fd945b2:::smithi10548823-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:50.035 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 188.21359331100001, 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079454387000000001, 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 151 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:50.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788977+0000", 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788976+0000", 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788980+0000", 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8999999999999999e-06 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788987+0000", 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5889999999999999e-06 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788994+0000", 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.669e-06 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835881+0000", 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.046886993000000002 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.835975+0000", 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4153000000000005e-05 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 1,7", 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836116+0000", 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014132000000000001 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.839088+0000", 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0029720910000000001 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841506+0000", 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024175999999999998 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868345+0000", 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.026839075 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868388+0000", 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3584999999999997e-05 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868431+0000", 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3044999999999999e-05 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:152 4.a 4:5fd945b2:::smithi10548823-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:50.041 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 188.21357215099999, 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079557584000000001, 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 152 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:50.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.788998+0000", 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789001+0000", 2024-04-20T02:53:50.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7870000000000002e-06 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789007+0000", 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4819999999999999e-06 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.789012+0000", 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0579999999999997e-06 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836344+0000", 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047331429000000001 2024-04-20T02:53:50.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.836388+0000", 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4595000000000002e-05 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868449+0000", 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.032060922999999998 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868500+0000", 2024-04-20T02:53:50.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1e-05 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.868556+0000", 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5702000000000003e-05 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:204 3.2 e29/27 3:51f7f66e:::smithi10548823-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 188.00600157, 2024-04-20T02:53:50.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074410591999999998, 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996569+0000", 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.996568+0000", 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000206+0000", 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003637894 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000218+0000", 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2074e-05 2024-04-20T02:53:50.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000230+0000", 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1737e-05 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000293+0000", 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2854000000000005e-05 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.000320+0000", 2024-04-20T02:53:50.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7223999999999999e-05 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070939+0000", 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070618467000000004 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.070979+0000", 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0661999999999998e-05 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:203 3.3 e29/21 3:e839d954:::smithi10548823-4 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.99180692499999, 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.073061940000000006, 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010763+0000", 2024-04-20T02:53:50.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.010762+0000", 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011156+0000", 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039334299999999999 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011173+0000", 2024-04-20T02:53:50.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7628000000000001e-05 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011185+0000", 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1542000000000001e-05 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011259+0000", 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4204999999999999e-05 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.011284+0000", 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.493e-05 2024-04-20T02:53:50.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083759+0000", 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.072475049999999999 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.083825+0000", 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6402000000000006e-05 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:205 3.0 e29/21 3:279c8ec1:::smithi10548823-6 low_tier:head v 29'1, mlcod=0'0)", 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.98583966800001, 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.069542860999999997, 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016731+0000", 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.016729+0000", 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019587+0000", 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002857493 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019601+0000", 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4239e-05 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019613+0000", 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2269000000000001e-05 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019665+0000", 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1499999999999998e-05 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.019689+0000", 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4641999999999999e-05 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086213+0000", 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066523141999999993 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.086274+0000", 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.0877e-05 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.058 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:201 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'2, mlcod=0'0)", 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.97743366500001, 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.070402822000000004, 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025137+0000", 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.025136+0000", 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027308+0000", 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.002172457 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027322+0000", 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3570999999999999e-05 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.027334+0000", 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2058999999999999e-05 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039809+0000", 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.012475689 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.039837+0000", 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7472999999999999e-05 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095491+0000", 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.055653962000000001 2024-04-20T02:53:50.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.095539+0000", 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:215 3.2 e29/27 3:6ba96d27:::smithi10548823-16 low_tier:head v 29'5, mlcod=0'0)", 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.87380162400001, 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.071063257000000005, 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.063 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128769+0000", 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.064 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.128767+0000", 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133592+0000", 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0048250389999999997 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133605+0000", 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639999999999999e-05 2024-04-20T02:53:50.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133618+0000", 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2989e-05 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133672+0000", 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4089999999999999e-05 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.133694+0000", 2024-04-20T02:53:50.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1484999999999998e-05 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199785+0000", 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066091105999999997 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.199832+0000", 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7268000000000002e-05 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:234 3.2 e29/27 3:43261aea:::smithi10548823-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'11, mlcod=29'6)", 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.643205314, 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10307089899999999, 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359365+0000", 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.359364+0000", 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362701+0000", 2024-04-20T02:53:50.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0033365230000000001 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362716+0000", 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5529999999999999e-05 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362724+0000", 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7530000000000001e-06 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362781+0000", 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6776e-05 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.362808+0000", 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7623999999999998e-05 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462376+0000", 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.099568059 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462436+0000", 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9617999999999997e-05 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:232 3.2 e29/27 3:77180fad:::smithi10548823-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'12, mlcod=29'6)", 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.616261165, 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.115627905, 2024-04-20T02:53:50.072 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386309+0000", 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.386308+0000", 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391388+0000", 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0050797170000000001 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391401+0000", 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3244999999999999e-05 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.391410+0000", 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.0790000000000007e-06 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400921+0000", 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0095106800000000005 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.400947+0000", 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5819000000000001e-05 2024-04-20T02:53:50.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501881+0000", 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.10093415 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501937+0000", 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6048999999999998e-05 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.076 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:235 3.2 e29/27 3:6e86a46f:::smithi10548823-36 low_tier:head v 29'13, mlcod=29'6)", 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.590902898, 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.090576580000000004, 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:50.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.411667+0000", 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417018+0000", 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053511699999999997 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417030+0000", 2024-04-20T02:53:50.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2247e-05 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.417039+0000", 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2010000000000005e-06 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426703+0000", 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0096638509999999993 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.426736+0000", 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.2851000000000001e-05 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502223+0000", 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.075487032999999995 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.502244+0000", 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1117000000000001e-05 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.080 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:236 3.2 e29/27 3:6a27cbd9:::smithi10548823-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'14, mlcod=29'6)", 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.56276245399999, 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.105353899, 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439808+0000", 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.439807+0000", 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444445+0000", 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0046385080000000004 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444461+0000", 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5843999999999999e-05 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.444471+0000", 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3179999999999996e-06 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462470+0000", 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017998920000000002 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.462493+0000", 2024-04-20T02:53:50.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3672e-05 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545114+0000", 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.082620822999999996 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545162+0000", 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7710000000000002e-05 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:248 3.3 e29/21 3:d54d2f71:::smithi10548823-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'7)", 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.52539384900001, 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.081537776000000006, 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477176+0000", 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477174+0000", 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999971 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478605+0000", 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001431267 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478657+0000", 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2252e-05 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478673+0000", 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5489e-05 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478727+0000", 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4669000000000001e-05 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.478752+0000", 2024-04-20T02:53:50.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4379000000000002e-05 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558668+0000", 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.079916635 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.558714+0000", 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5874000000000002e-05 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.088 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(osd.4.15:11 3.2 e29/27 3:6a76c5db:::smithi10548823-2 low_tier:head v 29'15, mlcod=29'10)", 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.525096926, 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.067707798999999999, 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477473+0000", 2024-04-20T02:53:50.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477472+0000", 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477493+0000", 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0571e-05 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477517+0000", 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4252000000000001e-05 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.477538+0000", 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1551000000000001e-05 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501972+0000", 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024433667999999999 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.501997+0000", 2024-04-20T02:53:50.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4459000000000001e-05 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545169+0000", 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043172031999999999 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.545181+0000", 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2639e-05 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:250 3.0 e29/21 3:19ac3882:::smithi10548823-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'8, mlcod=29'6)", 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.52001380799999, 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.080126771999999999, 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482557+0000", 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.482555+0000", 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484542+0000", 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0019865310000000001 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484557+0000", 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5014000000000001e-05 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484568+0000", 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1056000000000001e-05 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484621+0000", 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3347000000000003e-05 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.484651+0000", 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9913999999999999e-05 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562641+0000", 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.077989829999999996 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.562683+0000", 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2457999999999999e-05 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:246 3.0 e29/21 3:2e96546a:::smithi10548823-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 29'9, mlcod=29'6)", 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 187.49301124900001, 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084721467999999994, 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509559+0000", 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.509558+0000", 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515148+0000", 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.005590228 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515161+0000", 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2814e-05 2024-04-20T02:53:50.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.515169+0000", 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5369999999999997e-06 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519727+0000", 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004557485 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.519756+0000", 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9848999999999998e-05 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594251+0000", 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.074494311999999993 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:42.594281+0000", 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9748999999999999e-05 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.100 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:50.101 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-20T02:53:50.231 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:50.232 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:50.232 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:50.232 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:50.242 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-20T02:53:50.370 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:50.370 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:50.370 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:50.370 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:50.370 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:50.381 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-20T02:53:50.511 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:50.511 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:50.511 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:50.511 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:50.511 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.511 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e15)", 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 210.90692578900001, 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.32443517500000002, 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4201", 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.138:0/635542909", 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.512 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603915+0000", 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603913+0000", 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603922+0000", 2024-04-20T02:53:50.513 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.4909999999999994e-06 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603937+0000", 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4494e-05 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.603949+0000", 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1866e-05 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604038+0000", 2024-04-20T02:53:50.514 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9498000000000003e-05 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.604052+0000", 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3691e-05 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923607+0000", 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.31955536200000001 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.515 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.923811+0000", 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020418300000000001 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924028+0000", 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021634599999999999 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.924666+0000", 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00063776200000000005 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.516 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.517 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.926846+0000", 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0021803500000000002 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928259+0000", 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0014131580000000001 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928299+0000", 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0433999999999999e-05 2024-04-20T02:53:50.518 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.519 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.519 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.519 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:19.928350+0000", 2024-04-20T02:53:50.519 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0226000000000003e-05 2024-04-20T02:53:50.519 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.519 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.519 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.519 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:65 4.8 4:1afa5d2b:::smithi10548823-17:head [write 668912~636690,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 189.029756349, 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037726025000000003, 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 65 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.520 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481084+0000", 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481077+0000", 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999928 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.521 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481623+0000", 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000546018 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481642+0000", 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9298999999999999e-05 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481658+0000", 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5913000000000002e-05 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.522 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481716+0000", 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8108e-05 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.481901+0000", 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000184959 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.482090+0000", 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00018890700000000001 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.523 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.509854+0000", 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027764282000000001 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.516832+0000", 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0069778990000000001 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518718+0000", 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001886347 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.524 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518763+0000", 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4314999999999998e-05 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.518810+0000", 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7318000000000003e-05 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:165 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'4, mlcod=29'2)", 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:50.525 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 188.669970697, 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037442797, 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840870+0000", 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.526 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.840869+0000", 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841048+0000", 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000179202 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841060+0000", 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:50.527 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841068+0000", 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7870000000000002e-06 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841809+0000", 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074173900000000005 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.841831+0000", 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.1860000000000001e-05 2024-04-20T02:53:50.528 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878267+0000", 2024-04-20T02:53:50.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036435383000000002 2024-04-20T02:53:50.529 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.529 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.529 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.529 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.878312+0000", 2024-04-20T02:53:50.529 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5809000000000001e-05 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:166 4.5 e29/22 4:a9897a5e:::smithi10548823-42:head v 29'5, mlcod=29'2)", 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 188.65960709199999, 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039610919000000001, 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.530 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851233+0000", 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851232+0000", 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.531 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851638+0000", 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00040580999999999999 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851653+0000", 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4667999999999999e-05 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.851662+0000", 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3270000000000007e-06 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.532 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853363+0000", 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017009740000000001 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.853382+0000", 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9538000000000002e-05 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890793+0000", 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037410257000000002 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.533 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:50:41.890844+0000", 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1403999999999998e-05 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1437 4.8 4:1afa5d2b:::smithi10548823-17:head [write 467261~717275] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 170.369964388, 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041845932000000002, 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.534 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1437 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140876+0000", 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.535 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.140874+0000", 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141623+0000", 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00074936000000000002 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141642+0000", 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8338e-05 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.536 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141654+0000", 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2471e-05 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141717+0000", 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2476000000000003e-05 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.141818+0000", 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000101065 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.537 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.142008+0000", 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000190717 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182184+0000", 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040175178999999998 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182323+0000", 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013975500000000001 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.538 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182626+0000", 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00030255199999999999 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182675+0000", 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8671000000000002e-05 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.539 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182722+0000", 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7209000000000002e-05 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1438 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1847991~219270] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 170.36482727800001, 2024-04-20T02:53:50.540 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037840106999999998, 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1438 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.541 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146013+0000", 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146011+0000", 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146243+0000", 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023209200000000001 2024-04-20T02:53:50.542 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146262+0000", 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.8629000000000001e-05 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146272+0000", 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0530000000000001e-05 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.168942+0000", 2024-04-20T02:53:50.543 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.022669660000000001 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169018+0000", 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6402000000000005e-05 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.169179+0000", 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00016052800000000001 2024-04-20T02:53:50.544 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182313+0000", 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013134163000000001 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182330+0000", 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6671999999999999e-05 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183757+0000", 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001427109 2024-04-20T02:53:50.545 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183810+0000", 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.291e-05 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183853+0000", 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3288000000000003e-05 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.546 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1439 4.8 4:1afa5d2b:::smithi10548823-17:head [setxattr _header (56) in=7b,truncate 2067261] snapc d=[d,c] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 170.364554266, 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038307951, 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1439 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:50.547 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146286+0000", 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146285+0000", 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146292+0000", 2024-04-20T02:53:50.548 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7209999999999997e-06 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146300+0000", 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.3769999999999998e-06 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146310+0000", 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0217999999999999e-05 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181013+0000", 2024-04-20T02:53:50.549 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.034702853999999998 2024-04-20T02:53:50.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181131+0000", 2024-04-20T02:53:50.550 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000118767 2024-04-20T02:53:50.550 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.550 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.550 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 4,6", 2024-04-20T02:53:50.550 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.181342+0000", 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021065600000000001 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182853+0000", 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0015109089999999999 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.183898+0000", 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0010448129999999999 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:50.551 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184490+0000", 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00059210800000000004 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184538+0000", 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.7895999999999997e-05 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184594+0000", 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6267999999999997e-05 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.552 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1440 4.8 4:1afa5d2b:::smithi10548823-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 170.36452488099999, 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038506453000000003, 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1440 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.553 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146315+0000", 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:50.554 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146318+0000", 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6220000000000001e-06 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146324+0000", 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7920000000000004e-06 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.146329+0000", 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1170000000000003e-06 2024-04-20T02:53:50.555 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182360+0000", 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036030616000000001 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.182426+0000", 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6036000000000003e-05 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184627+0000", 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0022011800000000001 2024-04-20T02:53:50.556 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184711+0000", 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.3343999999999994e-05 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:00.184822+0000", 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000111209 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.557 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2000 4.10 e65/63 4:0b0a7da5:::smithi10548823-43:head v 65'72, mlcod=64'70)", 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 147.67719758199999, 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038161478999999998, 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833643+0000", 2024-04-20T02:53:50.558 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.833642+0000", 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834371+0000", 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072940199999999998 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834385+0000", 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4303e-05 2024-04-20T02:53:50.559 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834393+0000", 2024-04-20T02:53:50.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.7069999999999992e-06 2024-04-20T02:53:50.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.560 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.560 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.560 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834433+0000", 2024-04-20T02:53:50.560 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9555000000000003e-05 2024-04-20T02:53:50.560 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.834453+0000", 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0472999999999999e-05 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871762+0000", 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037308988000000001 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:22.871804+0000", 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2085000000000001e-05 2024-04-20T02:53:50.561 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2523 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'68, mlcod=86'65)", 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.22971736, 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043560513000000002, 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.562 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281123+0000", 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.563 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281522+0000", 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00039981800000000003 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281537+0000", 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4153e-05 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281546+0000", 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2960000000000004e-06 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.564 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281601+0000", 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4938999999999997e-05 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.281628+0000", 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.6784e-05 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324639+0000", 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043011075000000003 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.565 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.324683+0000", 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4817000000000001e-05 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:2524 4.8 e91/85 4:1afa5d2b:::smithi10548823-17:head v 91'69, mlcod=86'65)", 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.21244892599999, 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041875376999999998, 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.566 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298391+0000", 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298389+0000", 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.567 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298707+0000", 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00031746200000000002 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298722+0000", 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5537e-05 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298733+0000", 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0237999999999999e-05 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.568 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307149+0000", 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0084167879999999997 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.307180+0000", 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.0920000000000002e-05 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340235+0000", 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.033054618000000001 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.569 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340267+0000", 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1767000000000003e-05 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3145 4.15 e113/111 4:a9897a5e:::smithi10548823-42:head v 113'68, mlcod=0'0)", 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 99.039303130999997, 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045803218, 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.570 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.571 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.471537+0000", 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472014+0000", 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00047681900000000002 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472026+0000", 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2621e-05 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.572 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472035+0000", 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9479999999999997e-06 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472088+0000", 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2669e-05 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.472112+0000", 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4246000000000001e-05 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.573 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517287+0000", 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045174526 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:11.517340+0000", 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3665999999999997e-05 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3881 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'232, mlcod=141'228)", 2024-04-20T02:53:50.574 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.092180591000002, 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.050172267999999999, 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.575 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418660+0000", 2024-04-20T02:53:50.577 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.577 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.577 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.577 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.577 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.418659+0000", 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419496+0000", 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00083715799999999996 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419513+0000", 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7489000000000001e-05 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.578 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.419524+0000", 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0118e-05 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429881+0000", 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010357762 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.429905+0000", 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3546000000000001e-05 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468790+0000", 2024-04-20T02:53:50.579 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038884821 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468832+0000", 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.2341e-05 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3898 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'133, mlcod=133'129)", 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.040021619000001, 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038834664999999997, 2024-04-20T02:53:50.580 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470819+0000", 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.581 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.470817+0000", 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471051+0000", 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023384600000000001 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471066+0000", 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.4205e-05 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.582 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.471074+0000", 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.4090000000000003e-06 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472816+0000", 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0017419429999999999 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.472843+0000", 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7296000000000001e-05 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.583 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509592+0000", 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036748496999999998 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.509653+0000", 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.1707999999999999e-05 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3899 4.8 e146/114 4:1c68e542:::smithi10548823-8:head v 146'134, mlcod=133'129)", 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:50.584 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.022755183000001, 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.038218993999999999, 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488085+0000", 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.585 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488084+0000", 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488262+0000", 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017768600000000001 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488274+0000", 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2388000000000001e-05 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.586 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.488283+0000", 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.3840000000000003e-06 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489094+0000", 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00081045100000000001 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489119+0000", 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4751000000000001e-05 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.587 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526264+0000", 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037144912000000002 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526304+0000", 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0513000000000003e-05 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3891 4.7 e146/145 4:e2c0d65b:::smithi10548823-46:head v 146'105, mlcod=143'100)", 2024-04-20T02:53:50.588 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.021775087999998, 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.041132308999999999, 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.589 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489065+0000", 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489072+0000", 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9539999999999998e-06 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489082+0000", 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0329000000000001e-05 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.590 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489089+0000", 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4959999999999999e-06 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489814+0000", 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072423699999999997 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.489837+0000", 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.3218999999999999e-05 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.591 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530157+0000", 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040319762000000002 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530198+0000", 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0886000000000001e-05 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "MOSDPGPush(4.7 146/145 [PushOp(4:e2c0d65b:::smithi10548823-46:43, version: 143'96, data_included: [426944~820638,1468778~227938], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:e2c0d65b:::smithi10548823-46:43@143'96, size: 3311012, copy_subset: [0~3311012], clone_subset: {}, snapset: 45=[]:{3f=[3f,39],43=[41],45=[45]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1696716, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-04-20T02:53:50.592 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.020563736, 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039876652999999998, 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490277+0000", 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.593 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490276+0000", 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490785+0000", 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050941300000000001 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490804+0000", 2024-04-20T02:53:50.594 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9080999999999999e-05 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490823+0000", 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.855e-05 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498138+0000", 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.007315004 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498153+0000", 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5515e-05 2024-04-20T02:53:50.595 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530153+0000", 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031999845999999998 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4016 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'146, mlcod=147'143)", 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 62.248259490000002, 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054600045999999999, 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.596 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262581+0000", 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.597 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262580+0000", 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262733+0000", 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015289700000000001 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262746+0000", 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.375e-05 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.598 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262753+0000", 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6159999999999998e-06 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262817+0000", 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4202999999999995e-05 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262842+0000", 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.5296e-05 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.599 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317137+0000", 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054294664999999999 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317181+0000", 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.3819000000000002e-05 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4017 4.8 e149/114 4:1a8c7fb2:::smithi10548823-47:head v 149'147, mlcod=147'143)", 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 62.233212803999997, 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039585437000000001, 2024-04-20T02:53:50.600 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277628+0000", 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.601 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277626+0000", 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278058+0000", 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00043228299999999999 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278074+0000", 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5775000000000001e-05 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.602 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.278083+0000", 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2620000000000004e-06 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286849+0000", 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0087652210000000001 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.286881+0000", 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1919999999999999e-05 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.603 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317188+0000", 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.030307392999999998 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317213+0000", 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.4953999999999999e-05 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4340 4.8 e163/114 4:1a8c7fb2:::smithi10548823-47:head v 163'162, mlcod=159'158)", 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 47.598536418999998, 2024-04-20T02:53:50.604 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.04004311, 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912304+0000", 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:50.605 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912303+0000", 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912722+0000", 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00041911999999999999 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912738+0000", 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6294e-05 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.606 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912748+0000", 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6059999999999992e-06 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912840+0000", 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.2143000000000002e-05 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.912867+0000", 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.7121e-05 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.607 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952298+0000", 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039430999000000001 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:53:02.952347+0000", 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8990999999999999e-05 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.608 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:50.609 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:50.609 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:50.610 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-20T02:53:50.739 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:50.739 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:50.739 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:50.739 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:50.749 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-20T02:53:50.876 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:50.877 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:50.877 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:50.877 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:50.877 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:50.887 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-20T02:53:51.017 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [ 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:1659 3.2 e52/49 3:7a9f8f96:::599e580f52b6d0a71276ceab8388e31ecddf182120bb76f75e3501271632821c:head v 52'101, mlcod=48'99)", 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 161.56829125300001, 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052690951, 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.018 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.448638+0000", 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.019 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454082+0000", 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0054438510000000004 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454097+0000", 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5169e-05 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454107+0000", 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0395e-05 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.020 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454172+0000", 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5040000000000001e-05 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.454192+0000", 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.0034000000000002e-05 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501280+0000", 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.047087851 2024-04-20T02:53:51.021 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:09.501329+0000", 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8878000000000002e-05 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1903 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 152.51810848599999, 2024-04-20T02:53:51.022 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99715188200000004, 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1903 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.023 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498821+0000", 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498819+0000", 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999976 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498829+0000", 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0665e-05 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.024 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498843+0000", 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3611999999999999e-05 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498855+0000", 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1732999999999999e-05 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498974+0000", 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011921699999999999 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.025 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498984+0000", 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0387000000000001e-05 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495964+0000", 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99697954899999996 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495973+0000", 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.9150000000000002e-06 2024-04-20T02:53:51.026 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1904 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 152.51804303700001, 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99712115800000001, 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.027 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1904 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498886+0000", 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.028 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498885+0000", 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498894+0000", 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.0660000000000004e-06 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498900+0000", 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7889999999999997e-06 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.029 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498906+0000", 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8309999999999997e-06 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499006+0000", 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.9958999999999996e-05 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499012+0000", 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8279999999999999e-06 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.495999+0000", 2024-04-20T02:53:51.030 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99698698200000002 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496007+0000", 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5990000000000002e-06 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:1905 4.18 4.42a71638 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e61)", 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 152.518017539, 2024-04-20T02:53:51.031 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99722727200000005, 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "reached pg", 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 1905 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498912+0000", 2024-04-20T02:53:51.032 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.033 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.033 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.033 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498911+0000", 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498915+0000", 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.1599999999999998e-06 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498920+0000", 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4739999999999996e-06 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.034 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.498926+0000", 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2029999999999997e-06 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499030+0000", 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000104034 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:18.499036+0000", 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.2129999999999996e-06 2024-04-20T02:53:51.035 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496129+0000", 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.99709298599999996 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:19.496139+0000", 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6250000000000002e-06 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.036 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:2524 4.8 4:1afa5d2b:::smithi10548823-17:head [write 1723287~549075] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 122.73050805600001, 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.054106970999999997, 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 2524 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.037 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286421+0000", 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286419+0000", 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.038 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286924+0000", 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00050494600000000004 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286944+0000", 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.9833000000000001e-05 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.286957+0000", 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2901000000000001e-05 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.039 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298074+0000", 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.011117320999999999 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298151+0000", 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.6909999999999994e-05 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.298274+0000", 2024-04-20T02:53:51.040 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012279700000000001 2024-04-20T02:53:51.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.041 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.041 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.041 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.326267+0000", 2024-04-20T02:53:51.041 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.027993284 2024-04-20T02:53:51.041 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.331226+0000", 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0049590559999999999 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340442+0000", 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0092155740000000007 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340488+0000", 2024-04-20T02:53:51.042 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6208000000000002e-05 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:51:48.340528+0000", 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.0277000000000001e-05 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3637 4.1 e137/60 4:80c7a726:::smithi10548823-19:head v 137'124, mlcod=120'120)", 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 75.392520497000007, 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052640605, 2024-04-20T02:53:51.043 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624409+0000", 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.044 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624408+0000", 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624537+0000", 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012880199999999999 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624548+0000", 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0216e-05 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.045 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624556+0000", 2024-04-20T02:53:51.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.6179999999999995e-06 2024-04-20T02:53:51.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.046 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624603+0000", 2024-04-20T02:53:51.046 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6601999999999999e-05 2024-04-20T02:53:51.046 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.046 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.046 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.624632+0000", 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9277000000000001e-05 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.676999+0000", 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052366639999999999 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:35.677049+0000", 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0822999999999998e-05 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.047 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:3880 4.3 e146/60 4:c32272c0:::smithi10548823-41:head v 146'231, mlcod=141'228)", 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.610025516999997, 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.061291667000000001, 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.048 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406904+0000", 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.406903+0000", 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407359+0000", 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00045623700000000002 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.049 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407370+0000", 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1052e-05 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407377+0000", 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.1080000000000004e-06 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407429+0000", 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2033999999999997e-05 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.050 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.407458+0000", 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8050000000000001e-05 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468161+0000", 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060703628000000003 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.468196+0000", 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.4344999999999997e-05 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.051 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3888 4.7 4:e2c0d65b:::smithi10548823-46:head [write 413538~536345,stat] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.591472389000003, 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.059175945000000001, 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3888 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.052 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425457+0000", 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.425456+0000", 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426097+0000", 2024-04-20T02:53:51.053 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00064115099999999996 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426113+0000", 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.6118000000000001e-05 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426122+0000", 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.115e-06 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426179+0000", 2024-04-20T02:53:51.054 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6815999999999999e-05 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426316+0000", 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000136606 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.426499+0000", 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000183191 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443652+0000", 2024-04-20T02:53:51.055 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.017153471999999999 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484036+0000", 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040384097000000001 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484231+0000", 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000194811 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484363+0000", 2024-04-20T02:53:51.056 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00013146300000000001 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484426+0000", 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.3563000000000001e-05 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484633+0000", 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00020662299999999999 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.057 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3889 4.7 4:e2c0d65b:::smithi10548823-46:head [write 1395221~583781] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.584578414000006, 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.057918795000000002, 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3889 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.058 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432351+0000", 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.432350+0000", 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.059 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433490+0000", 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001140606 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433506+0000", 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.5828000000000002e-05 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.433515+0000", 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.8279999999999992e-06 2024-04-20T02:53:51.060 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443692+0000", 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.010177222 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.443839+0000", 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014662 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.444072+0000", 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023362200000000001 2024-04-20T02:53:51.061 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484114+0000", 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.040041641000000003 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484244+0000", 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012987200000000001 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484279+0000", 2024-04-20T02:53:51.062 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.5301e-05 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490055+0000", 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0057763459999999999 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490115+0000", 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.9320999999999997e-05 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490270+0000", 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00015498199999999999 2024-04-20T02:53:51.063 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3890 4.7 4:e2c0d65b:::smithi10548823-46:head [write 2556103~658617] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.576564313000006, 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.066878325000000002, 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3890 2024-04-20T02:53:51.064 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440365+0000", 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.065 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.440363+0000", 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441831+0000", 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.001467632 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441842+0000", 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0540000000000001e-05 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.066 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441850+0000", 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7260000000000008e-06 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463536+0000", 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.02168544 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463681+0000", 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000145086 2024-04-20T02:53:51.067 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.463941+0000", 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00025968600000000002 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484317+0000", 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.020376452 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490318+0000", 2024-04-20T02:53:51.068 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0060012490000000002 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490398+0000", 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9598000000000006e-05 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506988+0000", 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016589706999999999 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507066+0000", 2024-04-20T02:53:51.069 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.8299000000000004e-05 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507243+0000", 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000177503 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3891 4.7 4:e2c0d65b:::smithi10548823-46:head [setxattr _header (57) in=7b,truncate 3613538] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.575068803999997, 2024-04-20T02:53:51.070 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088590031, 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3891 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.071 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441861+0000", 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441859+0000", 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441864+0000", 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.8069999999999999e-06 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.072 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441869+0000", 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.0989999999999998e-06 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441875+0000", 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5820000000000001e-06 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484675+0000", 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.042799867999999998 2024-04-20T02:53:51.073 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.484817+0000", 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00014236899999999999 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,1,7", 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485048+0000", 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00023057199999999999 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.490359+0000", 2024-04-20T02:53:51.074 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0053116819999999999 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.506963+0000", 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.016603562999999998 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.507256+0000", 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00029288699999999998 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530316+0000", 2024-04-20T02:53:51.075 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.023060343000000001 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530356+0000", 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.9301999999999997e-05 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530451+0000", 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.5037999999999994e-05 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.076 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3892 4.7 4:e2c0d65b:::smithi10548823-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.575049116000002, 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.088912482000000001, 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3892 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.077 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441880+0000", 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.078 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441883+0000", 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9579999999999999e-06 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441889+0000", 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.7499999999999997e-06 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.441895+0000", 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4639999999999997e-06 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.079 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.080 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.080 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485431+0000", 2024-04-20T02:53:51.080 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043536321000000003 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.485537+0000", 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00010612599999999999 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530489+0000", 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044951996000000001 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.081 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530598+0000", 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000108937 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.530793+0000", 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00019441 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3898 4.8 4:1c68e542:::smithi10548823-8:head [write 447152~403196] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:51.082 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.550706032999997, 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.056917473000000003, 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3898 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.083 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466223+0000", 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466222+0000", 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.084 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466722+0000", 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00049941299999999998 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466734+0000", 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.2860999999999999e-05 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466743+0000", 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.7159999999999992e-06 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.085 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466809+0000", 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6165000000000001e-05 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.466928+0000", 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00011869199999999999 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.467158+0000", 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000230251 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.086 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.498338+0000", 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.03117996 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.522958+0000", 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.024619318000000001 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523002+0000", 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.4675999999999997e-05 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.087 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523082+0000", 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9803999999999997e-05 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523141+0000", 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.8767000000000002e-05 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:3899 4.8 4:1c68e542:::smithi10548823-8:head [write 1512185~587450] snapc 45=[45,41,3f,39] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 66.542778265999999, 2024-04-20T02:53:51.088 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060518556000000001, 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 3899 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.089 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474151+0000", 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474149+0000", 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474686+0000", 2024-04-20T02:53:51.090 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00053696199999999999 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474703+0000", 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.615e-05 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.474713+0000", 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0829000000000001e-05 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.091 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478533+0000", 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.003819687 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478640+0000", 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000106974 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.478766+0000", 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000125859 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.092 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.523803+0000", 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.045036616000000002 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.526432+0000", 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0026291489999999999 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534589+0000", 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0081572209999999992 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.093 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534623+0000", 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.3779000000000003e-05 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:44.534670+0000", 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6968999999999997e-05 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4016 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 665643~697786] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 62.755609501000002, 2024-04-20T02:53:51.094 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.060676251, 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4016 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.095 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261320+0000", 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.261318+0000", 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999986 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262081+0000", 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00076285299999999997 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.096 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262095+0000", 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3986e-05 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262118+0000", 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.2407000000000001e-05 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262164+0000", 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6248999999999997e-05 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262238+0000", 2024-04-20T02:53:51.097 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.4202000000000005e-05 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.262363+0000", 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012437799999999999 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314944+0000", 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.052581687000000002 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.098 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317407+0000", 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024628300000000001 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321887+0000", 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0044803020000000002 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321941+0000", 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.3769e-05 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.099 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.321996+0000", 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4988000000000003e-05 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4017 4.8 4:1a8c7fb2:::smithi10548823-47:head [write 2039745~753122] snapc 47=[47,45,3f] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 62.747979702999999, 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053146383999999998, 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.100 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4017 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268950+0000", 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.101 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.268948+0000", 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269678+0000", 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00072975799999999995 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269693+0000", 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.453e-05 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.102 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.269704+0000", 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.1623e-05 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277190+0000", 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0074859849999999997 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277276+0000", 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5809999999999993e-05 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.103 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 6,7", 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.277405+0000", 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012914600000000001 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.314952+0000", 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.037546931999999998 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.317441+0000", 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0024889840000000001 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.104 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322031+0000", 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.004589465 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322059+0000", 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.8730000000000001e-05 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:48.322096+0000", 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.6591999999999998e-05 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.105 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4074 4.11 4:8b91c759:::smithi10548823-5:head [write 1591248~408851] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.875012314000003, 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.053876146, 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4074 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.106 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141917+0000", 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.141916+0000", 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:51.107 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142740+0000", 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000823616 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142750+0000", 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 9.6150000000000003e-06 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142756+0000", 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.9419999999999998e-06 2024-04-20T02:53:51.108 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2061999999999997e-05 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142815+0000", 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.5050000000000004e-06 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174069+0000", 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.031253933999999997 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.109 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174193+0000", 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00012386 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.174438+0000", 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00024561400000000001 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187472+0000", 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.013033375999999999 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.110 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.188278+0000", 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00080595500000000002 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195551+0000", 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0072733340000000002 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195631+0000", 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 7.9882999999999994e-05 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.111 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.195793+0000", 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000162223 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4075 4.11 4:8b91c759:::smithi10548823-5:head [setxattr _header (56) in=7b,truncate 2000099] snapc 47=[47,45,3f] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.874161381999997, 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084064655000000002, 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.112 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4075 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142768+0000", 2024-04-20T02:53:51.113 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142767+0000", 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.999999 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142778+0000", 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0709e-05 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142783+0000", 2024-04-20T02:53:51.114 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.4539999999999998e-06 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142788+0000", 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.2850000000000002e-06 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142845+0000", 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.6919000000000002e-05 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142856+0000", 2024-04-20T02:53:51.115 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.0936e-05 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186635+0000", 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.043778897999999997 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186756+0000", 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000121087 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for subops from 0,6", 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.186967+0000", 2024-04-20T02:53:51.116 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00021064599999999999 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.189477+0000", 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0025104929999999999 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "op_commit", 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.190395+0000", 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00091717399999999998 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "sub_op_commit_rec", 2024-04-20T02:53:51.117 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226595+0000", 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.036200195999999997 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226661+0000", 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.6465999999999997e-05 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226833+0000", 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000171471 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.118 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_op(client.4431.0:4076 4.11 4:8b91c759:::smithi10548823-5:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 58.874120376, 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.084363853000000003, 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "started", 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "client_info": { 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "client": "client.4431", 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "client_addr": "172.21.15.105:0/1175622537", 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "tid": 4076 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.119 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142809+0000", 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142808+0000", 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999995 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.120 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142811+0000", 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 2.9229999999999998e-06 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142818+0000", 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 6.4420000000000003e-06 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142823+0000", 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.5199999999999997e-06 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.121 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142875+0000", 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 5.1759999999999999e-05 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for peered", 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.142880+0000", 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.6639999999999997e-06 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187624+0000", 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.044743949999999998 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.122 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "waiting for rw locks", 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.187726+0000", 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0001022 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.226882+0000", 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.039155862 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227003+0000", 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.000120715 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.123 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:52.227173+0000", 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.00017032899999999999 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: "description": "osd_repop(client.4431.0:4183 3.2 e154/49 3:4c962eae:::da630951d6f957584da594ce89c5cc3f1bab398b2577c643d7ab8ed57b8eb5bb:head v 154'234, mlcod=154'232)", 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: "initiated_at": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: "age": 57.363991443000003, 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.063121930000000007, 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: "type_data": { 2024-04-20T02:53:51.124 INFO:teuthology.orchestra.run.smithi138.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: "events": [ 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "initiated", 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "throttled", 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652938+0000", 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "header_read", 2024-04-20T02:53:51.125 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.652936+0000", 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4294967295.9999981 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "all_read", 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657053+0000", 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.0041172520000000001 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "dispatched", 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657067+0000", 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.3865e-05 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "queued_for_pg", 2024-04-20T02:53:51.126 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657076+0000", 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 8.5390000000000007e-06 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "reached_pg", 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657121+0000", 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 4.5114000000000002e-05 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "started", 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.657138+0000", 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 1.7600000000000001e-05 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "commit_sent", 2024-04-20T02:53:51.127 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716021+0000", 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 0.058882663000000002 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: }, 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: { 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: "event": "done", 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: "time": "2024-04-20T02:52:53.716060+0000", 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 3.8826999999999999e-05 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: } 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout: ] 2024-04-20T02:53:51.128 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:51.130 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-20T02:53:51.272 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:51.272 INFO:teuthology.orchestra.run.smithi105.stdout: "num_ops": 0 2024-04-20T02:53:51.272 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:51.283 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-20T02:53:51.410 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-20T02:53:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:51.413 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [], 2024-04-20T02:53:51.413 INFO:teuthology.orchestra.run.smithi105.stdout: "complaint_time": 30, 2024-04-20T02:53:51.413 INFO:teuthology.orchestra.run.smithi105.stdout: "num_blocked_ops": 0 2024-04-20T02:53:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:51.424 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-20T02:53:51.601 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:53:51.601 INFO:teuthology.orchestra.run.smithi105.stdout: "size": 20, 2024-04-20T02:53:51.601 INFO:teuthology.orchestra.run.smithi105.stdout: "duration": 600, 2024-04-20T02:53:51.601 INFO:teuthology.orchestra.run.smithi105.stdout: "ops": [] 2024-04-20T02:53:51.601 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:53:51.612 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-20T02:53:51.742 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:51.743 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:51.743 INFO:teuthology.orchestra.run.smithi138.stdout: "num_ops": 0 2024-04-20T02:53:51.743 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:51.754 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-20T02:53:51.882 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:51.882 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [], 2024-04-20T02:53:51.882 INFO:teuthology.orchestra.run.smithi138.stdout: "complaint_time": 30, 2024-04-20T02:53:51.882 INFO:teuthology.orchestra.run.smithi138.stdout: "num_blocked_ops": 0 2024-04-20T02:53:51.882 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:51.894 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-20T02:53:52.020 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2024-04-20T02:53:52.020 INFO:teuthology.orchestra.run.smithi138.stdout: "size": 20, 2024-04-20T02:53:52.021 INFO:teuthology.orchestra.run.smithi138.stdout: "duration": 600, 2024-04-20T02:53:52.021 INFO:teuthology.orchestra.run.smithi138.stdout: "ops": [] 2024-04-20T02:53:52.021 INFO:teuthology.orchestra.run.smithi138.stdout:} 2024-04-20T02:53:52.382 INFO:teuthology.orchestra.run.smithi105.stderr:noscrub is unset 2024-04-20T02:53:52.396 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-20T02:53:53.375 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-20T02:53:53.376 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-04-20T02:53:53.376 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-04-20T02:53:53.386 INFO:teuthology.orchestra.run.smithi105.stderr:nodeep-scrub is unset 2024-04-20T02:53:53.727 INFO:tasks.ceph.mon.a.smithi105.stderr:2024-04-20T02:53:53.727+0000 7f7f0b14c700 -1 mon.a@0(leader).osd e208 definitely_dead 0 2024-04-20T02:53:54.390 INFO:teuthology.orchestra.run.smithi105.stderr:marked in osd.4. 2024-04-20T02:53:54.404 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-04-20T02:53:54.404 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-20T02:53:54.405 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-04-20T02:53:54.405 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-04-20T02:53:54.747 INFO:tasks.ceph.mon.a.smithi105.stderr:2024-04-20T02:53:54.747+0000 7f7f0b14c700 -1 mon.a@0(leader).osd e209 definitely_dead 0 2024-04-20T02:53:55.397 INFO:teuthology.orchestra.run.smithi105.stderr:marked in osd.2. 2024-04-20T02:53:55.409 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-04-20T02:53:55.410 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-20T02:53:55.546 INFO:teuthology.orchestra.run.smithi105.stdout:mon.a: {} 2024-04-20T02:53:55.558 INFO:teuthology.orchestra.run.smithi105.stdout:mon.b: {} 2024-04-20T02:53:55.567 INFO:teuthology.orchestra.run.smithi105.stdout:mon.c: {} 2024-04-20T02:53:55.568 INFO:teuthology.orchestra.run.smithi105.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T02:53:55.569 INFO:teuthology.orchestra.run.smithi105.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T02:53:55.569 INFO:teuthology.orchestra.run.smithi105.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-20T02:53:55.580 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-20T02:53:56.156 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.0 to 1 (10000) 2024-04-20T02:53:56.169 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-20T02:53:57.158 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-20T02:53:57.171 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-20T02:53:58.160 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.1 to 1 (10000) 2024-04-20T02:53:58.175 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-20T02:53:59.158 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-20T02:53:59.172 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-20T02:54:00.423 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.3 to 1 (10000) 2024-04-20T02:54:00.437 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-20T02:54:01.154 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-20T02:54:01.167 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-20T02:54:02.428 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.6 to 1 (10000) 2024-04-20T02:54:02.442 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-20T02:54:03.429 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-20T02:54:03.444 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-20T02:54:04.438 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.7 to 1 (10000) 2024-04-20T02:54:04.450 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-04-20T02:54:05.446 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-20T02:54:05.457 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-20T02:54:06.448 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.5 to 1 (10000) 2024-04-20T02:54:06.462 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-20T02:54:07.456 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-20T02:54:07.468 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-20T02:54:08.465 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.2 to 1 (10000) 2024-04-20T02:54:08.478 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-20T02:54:09.467 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-20T02:54:09.480 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-20T02:54:10.475 INFO:teuthology.orchestra.run.smithi105.stderr:reweighted osd.4 to 1 (10000) 2024-04-20T02:54:10.489 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-20T02:54:11.162 INFO:teuthology.orchestra.run.smithi105.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-20T02:54:11.177 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-20T02:54:11.177 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-20T02:54:11.177 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-20T02:54:11.177 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-20T02:54:11.177 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-20T02:54:11.177 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T02:54:11.507 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:11.508 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":226,"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","created":"2024-04-20T02:50:07.356902+0000","modified":"2024-04-20T02:54:11.150296+0000","last_up_change":"2024-04-20T02:53:06.173366+0000","last_in_change":"2024-04-20T02:53:55.383050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T02:50:17.498569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-20T02:50:25.955448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-04-20T02:50:33.168857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":43330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":43330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6815","nonce":43330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":43330}]},"public_addr":"172.21.15.105:6812/43330","cluster_addr":"172.21.15.105:6813/43330","heartbeat_back_addr":"172.21.15.105:6815/43330","heartbeat_front_addr":"172.21.15.105:6814/43330","state":["exists","up"]},{"osd":1,"uuid":"c5ae8a64-fc7c-47ab-af6a-4b762a840645","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6805","nonce":43331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6807","nonce":43331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43331}]},"public_addr":"172.21.15.105:6804/43331","cluster_addr":"172.21.15.105:6805/43331","heartbeat_back_addr":"172.21.15.105:6807/43331","heartbeat_front_addr":"172.21.15.105:6806/43331","state":["exists","up"]},{"osd":2,"uuid":"336e852e-5fa6-4ac0-b053-d31ad40d7c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":121,"up_from":138,"up_thru":210,"down_at":122,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":66776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6801","nonce":66776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6803","nonce":66776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":66776}]},"public_addr":"172.21.15.105:6800/66776","cluster_addr":"172.21.15.105:6801/66776","heartbeat_back_addr":"172.21.15.105:6803/66776","heartbeat_front_addr":"172.21.15.105:6802/66776","state":["exists","up"]},{"osd":3,"uuid":"695cab3a-74d8-4046-ac61-2656dc0d893e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6809","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6811","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":43329}]},"public_addr":"172.21.15.105:6808/43329","cluster_addr":"172.21.15.105:6809/43329","heartbeat_back_addr":"172.21.15.105:6811/43329","heartbeat_front_addr":"172.21.15.105:6810/43329","state":["exists","up"]},{"osd":4,"uuid":"1d41da44-d894-40ac-acaa-aa0df15296c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":149,"up_from":167,"up_thru":212,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":66272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":66272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":66272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":66272}]},"public_addr":"172.21.15.138:6808/66272","cluster_addr":"172.21.15.138:6809/66272","heartbeat_back_addr":"172.21.15.138:6811/66272","heartbeat_front_addr":"172.21.15.138:6810/66272","state":["exists","up"]},{"osd":5,"uuid":"3e1c33fa-6c8c-4898-99bc-462ecef25f30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":25,"up_from":38,"up_thru":222,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":52590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":52590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":52590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":52590}]},"public_addr":"172.21.15.138:6802/52590","cluster_addr":"172.21.15.138:6804/52590","heartbeat_back_addr":"172.21.15.138:6807/52590","heartbeat_front_addr":"172.21.15.138:6806/52590","state":["exists","up"]},{"osd":6,"uuid":"fd1fc124-7cf8-4628-9076-1f27d6385676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":43329}]},"public_addr":"172.21.15.138:6800/43329","cluster_addr":"172.21.15.138:6801/43329","heartbeat_back_addr":"172.21.15.138:6805/43329","heartbeat_front_addr":"172.21.15.138:6803/43329","state":["exists","up"]},{"osd":7,"uuid":"53ae8193-5c3e-4698-aacd-df2ad1db9143","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":209,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":43351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6813","nonce":43351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6815","nonce":43351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":43351}]},"public_addr":"172.21.15.138:6812/43351","cluster_addr":"172.21.15.138:6813/43351","heartbeat_back_addr":"172.21.15.138:6815/43351","heartbeat_front_addr":"172.21.15.138:6814/43351","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:55.576317+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:49.338117+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:52:56.806719+0000","dead_epoch":122},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:54:10.018681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:50.396863+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:49.271276+0000","dead_epoch":26},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:44.475273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:32.093504+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.0","osds":[6,0,2]}],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.1","osds":[0,3,5]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T02:54:11.522 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-20T02:54:11.522 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T02:54:11.851 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:11.854 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":226,"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","created":"2024-04-20T02:50:07.356902+0000","modified":"2024-04-20T02:54:11.150296+0000","last_up_change":"2024-04-20T02:53:06.173366+0000","last_in_change":"2024-04-20T02:53:55.383050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T02:50:17.498569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-20T02:50:25.955448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-04-20T02:50:33.168857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":43330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":43330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6815","nonce":43330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":43330}]},"public_addr":"172.21.15.105:6812/43330","cluster_addr":"172.21.15.105:6813/43330","heartbeat_back_addr":"172.21.15.105:6815/43330","heartbeat_front_addr":"172.21.15.105:6814/43330","state":["exists","up"]},{"osd":1,"uuid":"c5ae8a64-fc7c-47ab-af6a-4b762a840645","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6805","nonce":43331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6807","nonce":43331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43331}]},"public_addr":"172.21.15.105:6804/43331","cluster_addr":"172.21.15.105:6805/43331","heartbeat_back_addr":"172.21.15.105:6807/43331","heartbeat_front_addr":"172.21.15.105:6806/43331","state":["exists","up"]},{"osd":2,"uuid":"336e852e-5fa6-4ac0-b053-d31ad40d7c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":121,"up_from":138,"up_thru":210,"down_at":122,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":66776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6801","nonce":66776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6803","nonce":66776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":66776}]},"public_addr":"172.21.15.105:6800/66776","cluster_addr":"172.21.15.105:6801/66776","heartbeat_back_addr":"172.21.15.105:6803/66776","heartbeat_front_addr":"172.21.15.105:6802/66776","state":["exists","up"]},{"osd":3,"uuid":"695cab3a-74d8-4046-ac61-2656dc0d893e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6809","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6811","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":43329}]},"public_addr":"172.21.15.105:6808/43329","cluster_addr":"172.21.15.105:6809/43329","heartbeat_back_addr":"172.21.15.105:6811/43329","heartbeat_front_addr":"172.21.15.105:6810/43329","state":["exists","up"]},{"osd":4,"uuid":"1d41da44-d894-40ac-acaa-aa0df15296c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":149,"up_from":167,"up_thru":212,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":66272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":66272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":66272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":66272}]},"public_addr":"172.21.15.138:6808/66272","cluster_addr":"172.21.15.138:6809/66272","heartbeat_back_addr":"172.21.15.138:6811/66272","heartbeat_front_addr":"172.21.15.138:6810/66272","state":["exists","up"]},{"osd":5,"uuid":"3e1c33fa-6c8c-4898-99bc-462ecef25f30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":25,"up_from":38,"up_thru":222,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":52590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":52590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":52590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":52590}]},"public_addr":"172.21.15.138:6802/52590","cluster_addr":"172.21.15.138:6804/52590","heartbeat_back_addr":"172.21.15.138:6807/52590","heartbeat_front_addr":"172.21.15.138:6806/52590","state":["exists","up"]},{"osd":6,"uuid":"fd1fc124-7cf8-4628-9076-1f27d6385676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":43329}]},"public_addr":"172.21.15.138:6800/43329","cluster_addr":"172.21.15.138:6801/43329","heartbeat_back_addr":"172.21.15.138:6805/43329","heartbeat_front_addr":"172.21.15.138:6803/43329","state":["exists","up"]},{"osd":7,"uuid":"53ae8193-5c3e-4698-aacd-df2ad1db9143","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":209,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":43351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6813","nonce":43351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6815","nonce":43351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":43351}]},"public_addr":"172.21.15.138:6812/43351","cluster_addr":"172.21.15.138:6813/43351","heartbeat_back_addr":"172.21.15.138:6815/43351","heartbeat_front_addr":"172.21.15.138:6814/43351","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:55.576317+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:49.338117+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:52:56.806719+0000","dead_epoch":122},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:54:10.018681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:50.396863+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:49.271276+0000","dead_epoch":26},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:44.475273+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:53:32.093504+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.0","osds":[6,0,2]}],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.1","osds":[0,3,5]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T02:54:11.866 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-20T02:54:11.866 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-20T02:54:11.867 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-20T02:54:11.867 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-20T02:54:11.867 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-20T02:54:11.868 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-20T02:54:11.868 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-20T02:54:11.868 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-20T02:54:12.157 INFO:teuthology.orchestra.run.smithi105.stdout:163208757289 2024-04-20T02:54:12.158 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-20T02:54:12.159 INFO:teuthology.orchestra.run.smithi105.stdout:55834574897 2024-04-20T02:54:12.160 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-20T02:54:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:55834574897 2024-04-20T02:54:12.161 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-20T02:54:12.168 INFO:teuthology.orchestra.run.smithi105.stdout:64424509489 2024-04-20T02:54:12.168 INFO:teuthology.orchestra.run.smithi105.stdout:64424509493 2024-04-20T02:54:12.169 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-20T02:54:12.169 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-20T02:54:12.172 INFO:teuthology.orchestra.run.smithi105.stdout:55834574899 2024-04-20T02:54:12.173 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-20T02:54:12.183 INFO:teuthology.orchestra.run.smithi105.stdout:592705486869 2024-04-20T02:54:12.183 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-20T02:54:12.190 INFO:teuthology.orchestra.run.smithi105.stdout:717259538447 2024-04-20T02:54:12.190 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-20T02:54:12.804 INFO:teuthology.orchestra.run.smithi105.stdout:163208757288 2024-04-20T02:54:12.804 INFO:teuthology.orchestra.run.smithi105.stdout:55834574896 2024-04-20T02:54:12.807 INFO:teuthology.orchestra.run.smithi105.stdout:55834574896 2024-04-20T02:54:12.829 INFO:tasks.ceph.ceph_manager.ceph:need seq 163208757289 got 163208757288 for osd.5 2024-04-20T02:54:12.834 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574896 for osd.1 2024-04-20T02:54:12.837 INFO:teuthology.orchestra.run.smithi105.stdout:64424509492 2024-04-20T02:54:12.838 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574896 for osd.6 2024-04-20T02:54:12.846 INFO:teuthology.orchestra.run.smithi105.stdout:592705486868 2024-04-20T02:54:12.850 INFO:teuthology.orchestra.run.smithi105.stdout:64424509488 2024-04-20T02:54:12.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509493 got 64424509492 for osd.7 2024-04-20T02:54:12.864 INFO:tasks.ceph.ceph_manager.ceph:need seq 592705486869 got 592705486868 for osd.2 2024-04-20T02:54:12.865 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509489 got 64424509488 for osd.0 2024-04-20T02:54:12.873 INFO:teuthology.orchestra.run.smithi105.stdout:55834574898 2024-04-20T02:54:12.890 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574899 got 55834574898 for osd.3 2024-04-20T02:54:12.904 INFO:teuthology.orchestra.run.smithi105.stdout:717259538446 2024-04-20T02:54:12.919 INFO:tasks.ceph.ceph_manager.ceph:need seq 717259538447 got 717259538446 for osd.4 2024-04-20T02:54:13.830 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-20T02:54:13.834 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-20T02:54:13.839 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-20T02:54:13.853 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-20T02:54:13.865 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-20T02:54:13.866 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-20T02:54:13.891 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-20T02:54:13.919 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-20T02:54:14.469 INFO:teuthology.orchestra.run.smithi105.stdout:64424509493 2024-04-20T02:54:14.473 INFO:teuthology.orchestra.run.smithi105.stdout:163208757290 2024-04-20T02:54:14.496 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509493 got 64424509493 for osd.7 2024-04-20T02:54:14.496 DEBUG:teuthology.parallel:result is None 2024-04-20T02:54:14.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 163208757289 got 163208757290 for osd.5 2024-04-20T02:54:14.497 DEBUG:teuthology.parallel:result is None 2024-04-20T02:54:14.500 INFO:teuthology.orchestra.run.smithi105.stdout:55834574898 2024-04-20T02:54:14.517 INFO:teuthology.orchestra.run.smithi105.stdout:592705486869 2024-04-20T02:54:14.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574898 for osd.6 2024-04-20T02:54:14.525 DEBUG:teuthology.parallel:result is None 2024-04-20T02:54:14.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 592705486869 got 592705486869 for osd.2 2024-04-20T02:54:14.534 DEBUG:teuthology.parallel:result is None 2024-04-20T02:54:14.537 INFO:teuthology.orchestra.run.smithi105.stdout:55834574898 2024-04-20T02:54:14.543 INFO:teuthology.orchestra.run.smithi105.stdout:55834574900 2024-04-20T02:54:14.546 INFO:teuthology.orchestra.run.smithi105.stdout:64424509489 2024-04-20T02:54:14.554 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574898 for osd.1 2024-04-20T02:54:14.554 DEBUG:teuthology.parallel:result is None 2024-04-20T02:54:14.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574899 got 55834574900 for osd.3 2024-04-20T02:54:14.559 DEBUG:teuthology.parallel:result is None 2024-04-20T02:54:14.569 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509489 got 64424509489 for osd.0 2024-04-20T02:54:14.569 DEBUG:teuthology.parallel:result is None 2024-04-20T02:54:14.591 INFO:teuthology.orchestra.run.smithi105.stdout:717259538447 2024-04-20T02:54:14.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 717259538447 got 717259538447 for osd.4 2024-04-20T02:54:14.606 DEBUG:teuthology.parallel:result is None 2024-04-20T02:54:14.606 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-20T02:54:14.606 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:14.971 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:14.973 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:14.985 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-04-20T02:54:13.489305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485512,"kb_used_data":209400,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489208,"statfs":{"total":773094113280,"available":772596948992,"internally_reserved":0,"allocated":214425600,"data_stored":217684560,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027356"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":189,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.843863+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:10.843863+0000","last_peered":"2024-04-20T02:54:10.843863+0000","last_clean":"2024-04-20T02:54:10.843863+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:10.843863+0000","last_undegraded":"2024-04-20T02:54:10.843863+0000","last_fullsized":"2024-04-20T02:54:10.843863+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":259,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325927+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:11.325927+0000","last_peered":"2024-04-20T02:54:11.325927+0000","last_clean":"2024-04-20T02:54:11.325927+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:11.325927+0000","last_undegraded":"2024-04-20T02:54:11.325927+0000","last_fullsized":"2024-04-20T02:54:11.325927+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":257,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-20T02:54:09.855344+0000","last_change":"2024-04-20T02:53:55.396705+0000","last_active":"2024-04-20T02:54:09.855344+0000","last_peered":"2024-04-20T02:54:09.855344+0000","last_clean":"2024-04-20T02:54:09.855344+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:09.855344+0000","last_undegraded":"2024-04-20T02:54:09.855344+0000","last_fullsized":"2024-04-20T02:54:09.855344+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:53.936957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:53.936957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.106940+0000","scrub_duration":0.0013534090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":647,"reported_epoch":226,"state":"remapped+peering","last_fresh":"2024-04-20T02:54:11.163227+0000","last_change":"2024-04-20T02:54:11.163427+0000","last_active":"2024-04-20T02:54:10.471720+0000","last_peered":"2024-04-20T02:54:10.355483+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:52:52.148432+0000","last_became_peered":"2024-04-20T02:52:52.148432+0000","last_unstale":"2024-04-20T02:54:11.163227+0000","last_undegraded":"2024-04-20T02:54:11.163227+0000","last_fullsized":"2024-04-20T02:54:11.163227+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.843872+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:10.843872+0000","last_peered":"2024-04-20T02:54:10.843872+0000","last_clean":"2024-04-20T02:54:10.843872+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:10.843872+0000","last_undegraded":"2024-04-20T02:54:10.843872+0000","last_fullsized":"2024-04-20T02:54:10.843872+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":642,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158350+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:11.158350+0000","last_peered":"2024-04-20T02:54:11.158350+0000","last_clean":"2024-04-20T02:54:11.158350+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:11.158350+0000","last_undegraded":"2024-04-20T02:54:11.158350+0000","last_fullsized":"2024-04-20T02:54:11.158350+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":391,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-20T02:54:09.855364+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:09.855364+0000","last_peered":"2024-04-20T02:54:09.855364+0000","last_clean":"2024-04-20T02:54:09.855364+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:09.855364+0000","last_undegraded":"2024-04-20T02:54:09.855364+0000","last_fullsized":"2024-04-20T02:54:09.855364+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":236,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158722+0000","last_change":"2024-04-20T02:53:56.572715+0000","last_active":"2024-04-20T02:54:11.158722+0000","last_peered":"2024-04-20T02:54:11.158722+0000","last_clean":"2024-04-20T02:54:11.158722+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:11.158722+0000","last_undegraded":"2024-04-20T02:54:11.158722+0000","last_fullsized":"2024-04-20T02:54:11.158722+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:16.451936+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":665,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325805+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:11.325805+0000","last_peered":"2024-04-20T02:54:11.325805+0000","last_clean":"2024-04-20T02:54:11.325805+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:11.325805+0000","last_undegraded":"2024-04-20T02:54:11.325805+0000","last_fullsized":"2024-04-20T02:54:11.325805+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.471286+0000","last_change":"2024-04-20T02:53:56.157458+0000","last_active":"2024-04-20T02:54:10.471286+0000","last_peered":"2024-04-20T02:54:10.471286+0000","last_clean":"2024-04-20T02:54:10.471286+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:10.471286+0000","last_undegraded":"2024-04-20T02:54:10.471286+0000","last_fullsized":"2024-04-20T02:54:10.471286+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:56.212430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.212430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.018625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":666,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.469871+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:10.469871+0000","last_peered":"2024-04-20T02:54:10.469871+0000","last_clean":"2024-04-20T02:54:10.469871+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:10.469871+0000","last_undegraded":"2024-04-20T02:54:10.469871+0000","last_fullsized":"2024-04-20T02:54:10.469871+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":203,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158712+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:11.158712+0000","last_peered":"2024-04-20T02:54:11.158712+0000","last_clean":"2024-04-20T02:54:11.158712+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:11.158712+0000","last_undegraded":"2024-04-20T02:54:11.158712+0000","last_fullsized":"2024-04-20T02:54:11.158712+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":241,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325876+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:11.325876+0000","last_peered":"2024-04-20T02:54:11.325876+0000","last_clean":"2024-04-20T02:54:11.325876+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:11.325876+0000","last_undegraded":"2024-04-20T02:54:11.325876+0000","last_fullsized":"2024-04-20T02:54:11.325876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509493,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37732,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334108,"statfs":{"total":96636764160,"available":96598126592,"internally_reserved":0,"allocated":2572288,"data_stored":1665308,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":2.3620000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.85199999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":1.083},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.67100000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":2.3279999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54100000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757290,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70372,"kb_used_data":34444,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301468,"statfs":{"total":96636764160,"available":96564703232,"internally_reserved":0,"allocated":35270656,"data_stored":36191852,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.47499999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.45900000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.42199999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.56899999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486869,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43332,"kb_used_data":15352,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328508,"statfs":{"total":96636764160,"available":96592392192,"internally_reserved":0,"allocated":15720448,"data_stored":15105572,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.35899999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.33100000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.31}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.42699999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.41399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54668,"kb_used_data":15352,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317172,"statfs":{"total":96636764160,"available":96580784128,"internally_reserved":0,"allocated":15720448,"data_stored":15105572,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.38100000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.432},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.54900000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.50600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69652,"kb_used_data":34432,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302188,"statfs":{"total":96636764160,"available":96565440512,"internally_reserved":0,"allocated":35258368,"data_stored":36187030,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.26900000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.40300000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.503}]}]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83432,"kb_used_data":48216,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288408,"statfs":{"total":96636764160,"available":96551329792,"internally_reserved":0,"allocated":49373184,"data_stored":51678602,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.502},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.58899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.70799999999999996}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.82199999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.72999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.745}]}]},{"osd":0,"up_from":15,"seq":64424509489,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77628,"kb_used_data":38312,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294212,"statfs":{"total":96636764160,"available":96557273088,"internally_reserved":0,"allocated":39231488,"data_stored":40633448,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.44}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.37}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.61199999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.69699999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.66100000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538447,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48696,"kb_used_data":20780,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323144,"statfs":{"total":96636764160,"available":96586899456,"internally_reserved":0,"allocated":21278720,"data_stored":21117176,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.35799999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.502}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61299999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.623},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:14.987 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:15.312 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:15.314 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:15.326 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-04-20T02:54:13.489305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485512,"kb_used_data":209400,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489208,"statfs":{"total":773094113280,"available":772596948992,"internally_reserved":0,"allocated":214425600,"data_stored":217684560,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027356"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":189,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.843863+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:10.843863+0000","last_peered":"2024-04-20T02:54:10.843863+0000","last_clean":"2024-04-20T02:54:10.843863+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:10.843863+0000","last_undegraded":"2024-04-20T02:54:10.843863+0000","last_fullsized":"2024-04-20T02:54:10.843863+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":259,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325927+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:11.325927+0000","last_peered":"2024-04-20T02:54:11.325927+0000","last_clean":"2024-04-20T02:54:11.325927+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:11.325927+0000","last_undegraded":"2024-04-20T02:54:11.325927+0000","last_fullsized":"2024-04-20T02:54:11.325927+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":257,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-20T02:54:09.855344+0000","last_change":"2024-04-20T02:53:55.396705+0000","last_active":"2024-04-20T02:54:09.855344+0000","last_peered":"2024-04-20T02:54:09.855344+0000","last_clean":"2024-04-20T02:54:09.855344+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:09.855344+0000","last_undegraded":"2024-04-20T02:54:09.855344+0000","last_fullsized":"2024-04-20T02:54:09.855344+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:53.936957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:53.936957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.106940+0000","scrub_duration":0.0013534090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":647,"reported_epoch":226,"state":"remapped+peering","last_fresh":"2024-04-20T02:54:11.163227+0000","last_change":"2024-04-20T02:54:11.163427+0000","last_active":"2024-04-20T02:54:10.471720+0000","last_peered":"2024-04-20T02:54:10.355483+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:52:52.148432+0000","last_became_peered":"2024-04-20T02:52:52.148432+0000","last_unstale":"2024-04-20T02:54:11.163227+0000","last_undegraded":"2024-04-20T02:54:11.163227+0000","last_fullsized":"2024-04-20T02:54:11.163227+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.843872+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:10.843872+0000","last_peered":"2024-04-20T02:54:10.843872+0000","last_clean":"2024-04-20T02:54:10.843872+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:10.843872+0000","last_undegraded":"2024-04-20T02:54:10.843872+0000","last_fullsized":"2024-04-20T02:54:10.843872+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":642,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158350+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:11.158350+0000","last_peered":"2024-04-20T02:54:11.158350+0000","last_clean":"2024-04-20T02:54:11.158350+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:11.158350+0000","last_undegraded":"2024-04-20T02:54:11.158350+0000","last_fullsized":"2024-04-20T02:54:11.158350+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":391,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-20T02:54:09.855364+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:09.855364+0000","last_peered":"2024-04-20T02:54:09.855364+0000","last_clean":"2024-04-20T02:54:09.855364+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:09.855364+0000","last_undegraded":"2024-04-20T02:54:09.855364+0000","last_fullsized":"2024-04-20T02:54:09.855364+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":236,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158722+0000","last_change":"2024-04-20T02:53:56.572715+0000","last_active":"2024-04-20T02:54:11.158722+0000","last_peered":"2024-04-20T02:54:11.158722+0000","last_clean":"2024-04-20T02:54:11.158722+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:11.158722+0000","last_undegraded":"2024-04-20T02:54:11.158722+0000","last_fullsized":"2024-04-20T02:54:11.158722+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:16.451936+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":665,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325805+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:11.325805+0000","last_peered":"2024-04-20T02:54:11.325805+0000","last_clean":"2024-04-20T02:54:11.325805+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:11.325805+0000","last_undegraded":"2024-04-20T02:54:11.325805+0000","last_fullsized":"2024-04-20T02:54:11.325805+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.471286+0000","last_change":"2024-04-20T02:53:56.157458+0000","last_active":"2024-04-20T02:54:10.471286+0000","last_peered":"2024-04-20T02:54:10.471286+0000","last_clean":"2024-04-20T02:54:10.471286+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:10.471286+0000","last_undegraded":"2024-04-20T02:54:10.471286+0000","last_fullsized":"2024-04-20T02:54:10.471286+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:52:56.212430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:52:56.212430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:12.018625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":666,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.469871+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:10.469871+0000","last_peered":"2024-04-20T02:54:10.469871+0000","last_clean":"2024-04-20T02:54:10.469871+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:10.469871+0000","last_undegraded":"2024-04-20T02:54:10.469871+0000","last_fullsized":"2024-04-20T02:54:10.469871+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":203,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158712+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:11.158712+0000","last_peered":"2024-04-20T02:54:11.158712+0000","last_clean":"2024-04-20T02:54:11.158712+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:11.158712+0000","last_undegraded":"2024-04-20T02:54:11.158712+0000","last_fullsized":"2024-04-20T02:54:11.158712+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":241,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325876+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:11.325876+0000","last_peered":"2024-04-20T02:54:11.325876+0000","last_clean":"2024-04-20T02:54:11.325876+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:11.325876+0000","last_undegraded":"2024-04-20T02:54:11.325876+0000","last_fullsized":"2024-04-20T02:54:11.325876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509493,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37732,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334108,"statfs":{"total":96636764160,"available":96598126592,"internally_reserved":0,"allocated":2572288,"data_stored":1665308,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":2.3620000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.85199999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":1.083},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.67100000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":2.3279999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54100000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757290,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70372,"kb_used_data":34444,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301468,"statfs":{"total":96636764160,"available":96564703232,"internally_reserved":0,"allocated":35270656,"data_stored":36191852,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.47499999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.45900000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.42199999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.56899999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486869,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43332,"kb_used_data":15352,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328508,"statfs":{"total":96636764160,"available":96592392192,"internally_reserved":0,"allocated":15720448,"data_stored":15105572,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.35899999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.33100000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.31}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.42699999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.41399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54668,"kb_used_data":15352,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317172,"statfs":{"total":96636764160,"available":96580784128,"internally_reserved":0,"allocated":15720448,"data_stored":15105572,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.38100000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.432},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.54900000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.50600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69652,"kb_used_data":34432,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302188,"statfs":{"total":96636764160,"available":96565440512,"internally_reserved":0,"allocated":35258368,"data_stored":36187030,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.26900000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.40300000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.503}]}]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83432,"kb_used_data":48216,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288408,"statfs":{"total":96636764160,"available":96551329792,"internally_reserved":0,"allocated":49373184,"data_stored":51678602,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.502},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.58899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.70799999999999996}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.82199999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.72999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.745}]}]},{"osd":0,"up_from":15,"seq":64424509489,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77628,"kb_used_data":38312,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294212,"statfs":{"total":96636764160,"available":96557273088,"internally_reserved":0,"allocated":39231488,"data_stored":40633448,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.44}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.37}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.42099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.61199999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.69699999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.66100000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538447,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48696,"kb_used_data":20780,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323144,"statfs":{"total":96636764160,"available":96586899456,"internally_reserved":0,"allocated":21278720,"data_stored":21117176,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.35799999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.502}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61299999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.623},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:15.328 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:54:15.738 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:15.738 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":248,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":227,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497217536,"bytes_avail":772596895744,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:54:15.752 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:54:15.753 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:16.074 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:16.075 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:16.087 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-04-20T02:54:15.489805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485564,"kb_used_data":209448,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489156,"statfs":{"total":773094113280,"available":772596895744,"internally_reserved":0,"allocated":214474752,"data_stored":217703848,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001368"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":189,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.843863+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:10.843863+0000","last_peered":"2024-04-20T02:54:10.843863+0000","last_clean":"2024-04-20T02:54:10.843863+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:10.843863+0000","last_undegraded":"2024-04-20T02:54:10.843863+0000","last_fullsized":"2024-04-20T02:54:10.843863+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":259,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325927+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:11.325927+0000","last_peered":"2024-04-20T02:54:11.325927+0000","last_clean":"2024-04-20T02:54:11.325927+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:11.325927+0000","last_undegraded":"2024-04-20T02:54:11.325927+0000","last_fullsized":"2024-04-20T02:54:11.325927+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775097+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775097+0000","last_peered":"2024-04-20T02:54:12.775097+0000","last_clean":"2024-04-20T02:54:12.775097+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775097+0000","last_undegraded":"2024-04-20T02:54:12.775097+0000","last_fullsized":"2024-04-20T02:54:12.775097+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":658,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177819+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177819+0000","last_peered":"2024-04-20T02:54:12.177819+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177819+0000","last_undegraded":"2024-04-20T02:54:12.177819+0000","last_fullsized":"2024-04-20T02:54:12.177819+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.843872+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:10.843872+0000","last_peered":"2024-04-20T02:54:10.843872+0000","last_clean":"2024-04-20T02:54:10.843872+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:10.843872+0000","last_undegraded":"2024-04-20T02:54:10.843872+0000","last_fullsized":"2024-04-20T02:54:10.843872+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":642,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158350+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:11.158350+0000","last_peered":"2024-04-20T02:54:11.158350+0000","last_clean":"2024-04-20T02:54:11.158350+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:11.158350+0000","last_undegraded":"2024-04-20T02:54:11.158350+0000","last_fullsized":"2024-04-20T02:54:11.158350+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":392,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.326784+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:11.326784+0000","last_peered":"2024-04-20T02:54:11.326784+0000","last_clean":"2024-04-20T02:54:11.326784+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:11.326784+0000","last_undegraded":"2024-04-20T02:54:11.326784+0000","last_fullsized":"2024-04-20T02:54:11.326784+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":236,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158722+0000","last_change":"2024-04-20T02:53:56.572715+0000","last_active":"2024-04-20T02:54:11.158722+0000","last_peered":"2024-04-20T02:54:11.158722+0000","last_clean":"2024-04-20T02:54:11.158722+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:11.158722+0000","last_undegraded":"2024-04-20T02:54:11.158722+0000","last_fullsized":"2024-04-20T02:54:11.158722+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:16.451936+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":665,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325805+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:11.325805+0000","last_peered":"2024-04-20T02:54:11.325805+0000","last_clean":"2024-04-20T02:54:11.325805+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:11.325805+0000","last_undegraded":"2024-04-20T02:54:11.325805+0000","last_fullsized":"2024-04-20T02:54:11.325805+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795558+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795558+0000","last_peered":"2024-04-20T02:54:12.795558+0000","last_clean":"2024-04-20T02:54:12.795558+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795558+0000","last_undegraded":"2024-04-20T02:54:12.795558+0000","last_fullsized":"2024-04-20T02:54:12.795558+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":667,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.156294+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:11.156294+0000","last_peered":"2024-04-20T02:54:11.156294+0000","last_clean":"2024-04-20T02:54:11.156294+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:11.156294+0000","last_undegraded":"2024-04-20T02:54:11.156294+0000","last_fullsized":"2024-04-20T02:54:11.156294+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":203,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158712+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:11.158712+0000","last_peered":"2024-04-20T02:54:11.158712+0000","last_clean":"2024-04-20T02:54:11.158712+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:11.158712+0000","last_undegraded":"2024-04-20T02:54:11.158712+0000","last_fullsized":"2024-04-20T02:54:11.158712+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":241,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325876+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:11.325876+0000","last_peered":"2024-04-20T02:54:11.325876+0000","last_clean":"2024-04-20T02:54:11.325876+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:11.325876+0000","last_undegraded":"2024-04-20T02:54:11.325876+0000","last_fullsized":"2024-04-20T02:54:11.325876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509494,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37736,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334104,"statfs":{"total":96636764160,"available":96598122496,"internally_reserved":0,"allocated":2584576,"data_stored":1670130,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":2.3620000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.85199999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":1.083},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.67100000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":2.3279999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54100000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757290,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70372,"kb_used_data":34444,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301468,"statfs":{"total":96636764160,"available":96564703232,"internally_reserved":0,"allocated":35270656,"data_stored":36191852,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.47499999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.45900000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.42199999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.56899999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43348,"kb_used_data":15364,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328492,"statfs":{"total":96636764160,"available":96592375808,"internally_reserved":0,"allocated":15732736,"data_stored":15110394,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.34699999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.318}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.254}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.505}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.57799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54668,"kb_used_data":15352,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317172,"statfs":{"total":96636764160,"available":96580784128,"internally_reserved":0,"allocated":15720448,"data_stored":15105572,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.38100000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.432},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.54900000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.50600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69652,"kb_used_data":34432,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302188,"statfs":{"total":96636764160,"available":96565440512,"internally_reserved":0,"allocated":35258368,"data_stored":36187030,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.26900000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.40300000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.503}]}]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83432,"kb_used_data":48216,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288408,"statfs":{"total":96636764160,"available":96551329792,"internally_reserved":0,"allocated":49373184,"data_stored":51678602,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.502},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.58899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.70799999999999996}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.82199999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.72999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.745}]}]},{"osd":0,"up_from":15,"seq":64424509490,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39243776,"data_stored":40638270,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":1.149},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":1.2629999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":1.29},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.1899999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":1.327},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":1.556}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":1.4390000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.238}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":1.179},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.5309999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.3029999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.66100000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538448,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48712,"kb_used_data":20792,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323128,"statfs":{"total":96636764160,"available":96586883072,"internally_reserved":0,"allocated":21291008,"data_stored":21121998,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.35799999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.502}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61299999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.623},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:19.089 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:19.431 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:19.432 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:19.443 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-04-20T02:54:17.490430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485564,"kb_used_data":209448,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489156,"statfs":{"total":773094113280,"available":772596895744,"internally_reserved":0,"allocated":214474752,"data_stored":217703848,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.335300"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":189,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.843863+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:10.843863+0000","last_peered":"2024-04-20T02:54:10.843863+0000","last_clean":"2024-04-20T02:54:10.843863+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:10.843863+0000","last_undegraded":"2024-04-20T02:54:10.843863+0000","last_fullsized":"2024-04-20T02:54:10.843863+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":259,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325927+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:11.325927+0000","last_peered":"2024-04-20T02:54:11.325927+0000","last_clean":"2024-04-20T02:54:11.325927+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:11.325927+0000","last_undegraded":"2024-04-20T02:54:11.325927+0000","last_fullsized":"2024-04-20T02:54:11.325927+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775097+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775097+0000","last_peered":"2024-04-20T02:54:12.775097+0000","last_clean":"2024-04-20T02:54:12.775097+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775097+0000","last_undegraded":"2024-04-20T02:54:12.775097+0000","last_fullsized":"2024-04-20T02:54:12.775097+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":658,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177819+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177819+0000","last_peered":"2024-04-20T02:54:12.177819+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177819+0000","last_undegraded":"2024-04-20T02:54:12.177819+0000","last_fullsized":"2024-04-20T02:54:12.177819+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-20T02:54:10.843872+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:10.843872+0000","last_peered":"2024-04-20T02:54:10.843872+0000","last_clean":"2024-04-20T02:54:10.843872+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:10.843872+0000","last_undegraded":"2024-04-20T02:54:10.843872+0000","last_fullsized":"2024-04-20T02:54:10.843872+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":642,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158350+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:11.158350+0000","last_peered":"2024-04-20T02:54:11.158350+0000","last_clean":"2024-04-20T02:54:11.158350+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:11.158350+0000","last_undegraded":"2024-04-20T02:54:11.158350+0000","last_fullsized":"2024-04-20T02:54:11.158350+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":392,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.326784+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:11.326784+0000","last_peered":"2024-04-20T02:54:11.326784+0000","last_clean":"2024-04-20T02:54:11.326784+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:11.326784+0000","last_undegraded":"2024-04-20T02:54:11.326784+0000","last_fullsized":"2024-04-20T02:54:11.326784+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":236,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158722+0000","last_change":"2024-04-20T02:53:56.572715+0000","last_active":"2024-04-20T02:54:11.158722+0000","last_peered":"2024-04-20T02:54:11.158722+0000","last_clean":"2024-04-20T02:54:11.158722+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:11.158722+0000","last_undegraded":"2024-04-20T02:54:11.158722+0000","last_fullsized":"2024-04-20T02:54:11.158722+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:08.364015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.364015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:16.451936+0000","scrub_duration":0.0012668860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":665,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325805+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:11.325805+0000","last_peered":"2024-04-20T02:54:11.325805+0000","last_clean":"2024-04-20T02:54:11.325805+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:11.325805+0000","last_undegraded":"2024-04-20T02:54:11.325805+0000","last_fullsized":"2024-04-20T02:54:11.325805+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795558+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795558+0000","last_peered":"2024-04-20T02:54:12.795558+0000","last_clean":"2024-04-20T02:54:12.795558+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795558+0000","last_undegraded":"2024-04-20T02:54:12.795558+0000","last_fullsized":"2024-04-20T02:54:12.795558+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":667,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.156294+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:11.156294+0000","last_peered":"2024-04-20T02:54:11.156294+0000","last_clean":"2024-04-20T02:54:11.156294+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:11.156294+0000","last_undegraded":"2024-04-20T02:54:11.156294+0000","last_fullsized":"2024-04-20T02:54:11.156294+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":203,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.158712+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:11.158712+0000","last_peered":"2024-04-20T02:54:11.158712+0000","last_clean":"2024-04-20T02:54:11.158712+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:11.158712+0000","last_undegraded":"2024-04-20T02:54:11.158712+0000","last_fullsized":"2024-04-20T02:54:11.158712+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":241,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.325876+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:11.325876+0000","last_peered":"2024-04-20T02:54:11.325876+0000","last_clean":"2024-04-20T02:54:11.325876+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:11.325876+0000","last_undegraded":"2024-04-20T02:54:11.325876+0000","last_fullsized":"2024-04-20T02:54:11.325876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509494,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37736,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334104,"statfs":{"total":96636764160,"available":96598122496,"internally_reserved":0,"allocated":2584576,"data_stored":1670130,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":2.3620000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.85199999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":1.083},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.67100000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":2.3279999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54100000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757290,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70372,"kb_used_data":34444,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301468,"statfs":{"total":96636764160,"available":96564703232,"internally_reserved":0,"allocated":35270656,"data_stored":36191852,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59299999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.47499999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.45900000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.42199999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.56899999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43348,"kb_used_data":15364,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328492,"statfs":{"total":96636764160,"available":96592375808,"internally_reserved":0,"allocated":15732736,"data_stored":15110394,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.34699999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.318}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.254}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.505}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.57799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54668,"kb_used_data":15352,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317172,"statfs":{"total":96636764160,"available":96580784128,"internally_reserved":0,"allocated":15720448,"data_stored":15105572,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.38100000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.432},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.54900000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.50600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69652,"kb_used_data":34432,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302188,"statfs":{"total":96636764160,"available":96565440512,"internally_reserved":0,"allocated":35258368,"data_stored":36187030,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.26900000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.40300000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.503}]}]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83432,"kb_used_data":48216,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288408,"statfs":{"total":96636764160,"available":96551329792,"internally_reserved":0,"allocated":49373184,"data_stored":51678602,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.502},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.58899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.70799999999999996}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.82199999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.72999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.745}]}]},{"osd":0,"up_from":15,"seq":64424509490,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39243776,"data_stored":40638270,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":1.149},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":1.2629999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":1.29},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.1899999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":1.327},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":1.556}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":1.4390000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.238}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":1.179},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.5309999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.3029999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.66100000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538448,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48712,"kb_used_data":20792,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323128,"statfs":{"total":96636764160,"available":96586883072,"internally_reserved":0,"allocated":21291008,"data_stored":21121998,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.35799999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.502}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61299999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.623},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:19.446 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-20T02:54:19.446 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-20T02:54:19.792 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-20T02:54:19.803 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-20T02:54:19.803 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:20.131 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:20.133 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:20.144 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-04-20T02:54:19.490914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485604,"kb_used_data":209520,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489116,"statfs":{"total":773094113280,"available":772596854784,"internally_reserved":0,"allocated":214548480,"data_stored":217732606,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002060"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775097+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775097+0000","last_peered":"2024-04-20T02:54:12.775097+0000","last_clean":"2024-04-20T02:54:12.775097+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775097+0000","last_undegraded":"2024-04-20T02:54:12.775097+0000","last_fullsized":"2024-04-20T02:54:12.775097+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":658,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177819+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177819+0000","last_peered":"2024-04-20T02:54:12.177819+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177819+0000","last_undegraded":"2024-04-20T02:54:12.177819+0000","last_fullsized":"2024-04-20T02:54:12.177819+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":392,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.326784+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:11.326784+0000","last_peered":"2024-04-20T02:54:11.326784+0000","last_clean":"2024-04-20T02:54:11.326784+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:11.326784+0000","last_undegraded":"2024-04-20T02:54:11.326784+0000","last_fullsized":"2024-04-20T02:54:11.326784+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795558+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795558+0000","last_peered":"2024-04-20T02:54:12.795558+0000","last_clean":"2024-04-20T02:54:12.795558+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795558+0000","last_undegraded":"2024-04-20T02:54:12.795558+0000","last_fullsized":"2024-04-20T02:54:12.795558+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":667,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.156294+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:11.156294+0000","last_peered":"2024-04-20T02:54:11.156294+0000","last_clean":"2024-04-20T02:54:11.156294+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:11.156294+0000","last_undegraded":"2024-04-20T02:54:11.156294+0000","last_fullsized":"2024-04-20T02:54:11.156294+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509494,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37736,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334104,"statfs":{"total":96636764160,"available":96598122496,"internally_reserved":0,"allocated":2584576,"data_stored":1670130,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":2.3620000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.85199999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":1.083},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.67100000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":2.3279999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54100000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70372,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301468,"statfs":{"total":96636764160,"available":96564703232,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.76200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.80400000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.71299999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.624}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.871},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.72799999999999998}]}]},{"osd":2,"up_from":138,"seq":592705486870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43348,"kb_used_data":15364,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328492,"statfs":{"total":96636764160,"available":96592375808,"internally_reserved":0,"allocated":15732736,"data_stored":15110394,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.34699999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.318}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.254}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.505}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.57799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54672,"kb_used_data":15364,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317168,"statfs":{"total":96636764160,"available":96580780032,"internally_reserved":0,"allocated":15732736,"data_stored":15110394,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.311}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.64200000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.73499999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.32200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.25600000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67700000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.621},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.70999999999999996}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.73699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.495},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.67100000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509490,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39243776,"data_stored":40638270,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":1.149},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":1.2629999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":1.29},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.1899999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":1.327},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":1.556}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":1.4390000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.238}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":1.179},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.5309999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.3029999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.66100000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538448,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48712,"kb_used_data":20792,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323128,"statfs":{"total":96636764160,"available":96586883072,"internally_reserved":0,"allocated":21291008,"data_stored":21121998,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.35799999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.502}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61299999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.623},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:20.146 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:20.473 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:20.475 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:20.487 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-04-20T02:54:19.490914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485604,"kb_used_data":209520,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489116,"statfs":{"total":773094113280,"available":772596854784,"internally_reserved":0,"allocated":214548480,"data_stored":217732606,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002060"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775097+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775097+0000","last_peered":"2024-04-20T02:54:12.775097+0000","last_clean":"2024-04-20T02:54:12.775097+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775097+0000","last_undegraded":"2024-04-20T02:54:12.775097+0000","last_fullsized":"2024-04-20T02:54:12.775097+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":658,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177819+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177819+0000","last_peered":"2024-04-20T02:54:12.177819+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177819+0000","last_undegraded":"2024-04-20T02:54:12.177819+0000","last_fullsized":"2024-04-20T02:54:12.177819+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":392,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.326784+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:11.326784+0000","last_peered":"2024-04-20T02:54:11.326784+0000","last_clean":"2024-04-20T02:54:11.326784+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:11.326784+0000","last_undegraded":"2024-04-20T02:54:11.326784+0000","last_fullsized":"2024-04-20T02:54:11.326784+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795558+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795558+0000","last_peered":"2024-04-20T02:54:12.795558+0000","last_clean":"2024-04-20T02:54:12.795558+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795558+0000","last_undegraded":"2024-04-20T02:54:12.795558+0000","last_fullsized":"2024-04-20T02:54:12.795558+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":667,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.156294+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:11.156294+0000","last_peered":"2024-04-20T02:54:11.156294+0000","last_clean":"2024-04-20T02:54:11.156294+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:11.156294+0000","last_undegraded":"2024-04-20T02:54:11.156294+0000","last_fullsized":"2024-04-20T02:54:11.156294+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509494,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37736,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334104,"statfs":{"total":96636764160,"available":96598122496,"internally_reserved":0,"allocated":2584576,"data_stored":1670130,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":2.3620000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.85199999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":1.083},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.67100000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":2.3279999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54100000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70372,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301468,"statfs":{"total":96636764160,"available":96564703232,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.76200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.80400000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.71299999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.624}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.871},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.72799999999999998}]}]},{"osd":2,"up_from":138,"seq":592705486870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43348,"kb_used_data":15364,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328492,"statfs":{"total":96636764160,"available":96592375808,"internally_reserved":0,"allocated":15732736,"data_stored":15110394,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.34699999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.318}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.254}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.505}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.57799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54672,"kb_used_data":15364,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317168,"statfs":{"total":96636764160,"available":96580780032,"internally_reserved":0,"allocated":15732736,"data_stored":15110394,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.311}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.64200000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.73499999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.32200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.25600000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67700000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.621},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.70999999999999996}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.73699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.495},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.67100000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509490,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39243776,"data_stored":40638270,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":1.149},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":1.2629999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":1.29},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.1899999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":1.327},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":1.556}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":1.4390000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.238}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":1.179},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.5309999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.3029999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.66100000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538448,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48712,"kb_used_data":20792,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323128,"statfs":{"total":96636764160,"available":96586883072,"internally_reserved":0,"allocated":21291008,"data_stored":21121998,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.35799999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.502}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61299999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.623},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:20.489 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:20.816 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:20.818 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:20.829 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-04-20T02:54:19.490914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485604,"kb_used_data":209520,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489116,"statfs":{"total":773094113280,"available":772596854784,"internally_reserved":0,"allocated":214548480,"data_stored":217732606,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002060"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775097+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775097+0000","last_peered":"2024-04-20T02:54:12.775097+0000","last_clean":"2024-04-20T02:54:12.775097+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775097+0000","last_undegraded":"2024-04-20T02:54:12.775097+0000","last_fullsized":"2024-04-20T02:54:12.775097+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":658,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177819+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177819+0000","last_peered":"2024-04-20T02:54:12.177819+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177819+0000","last_undegraded":"2024-04-20T02:54:12.177819+0000","last_fullsized":"2024-04-20T02:54:12.177819+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":392,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.326784+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:11.326784+0000","last_peered":"2024-04-20T02:54:11.326784+0000","last_clean":"2024-04-20T02:54:11.326784+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:11.326784+0000","last_undegraded":"2024-04-20T02:54:11.326784+0000","last_fullsized":"2024-04-20T02:54:11.326784+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795558+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795558+0000","last_peered":"2024-04-20T02:54:12.795558+0000","last_clean":"2024-04-20T02:54:12.795558+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795558+0000","last_undegraded":"2024-04-20T02:54:12.795558+0000","last_fullsized":"2024-04-20T02:54:12.795558+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":667,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-04-20T02:54:11.156294+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:11.156294+0000","last_peered":"2024-04-20T02:54:11.156294+0000","last_clean":"2024-04-20T02:54:11.156294+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:11.156294+0000","last_undegraded":"2024-04-20T02:54:11.156294+0000","last_fullsized":"2024-04-20T02:54:11.156294+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509494,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37736,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334104,"statfs":{"total":96636764160,"available":96598122496,"internally_reserved":0,"allocated":2584576,"data_stored":1670130,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":2.3620000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.85199999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":1.083},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.67100000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":2.3279999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.40500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.501},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54100000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70372,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301468,"statfs":{"total":96636764160,"available":96564703232,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.76200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.80400000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.71299999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.624}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.871},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.72799999999999998}]}]},{"osd":2,"up_from":138,"seq":592705486870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43348,"kb_used_data":15364,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328492,"statfs":{"total":96636764160,"available":96592375808,"internally_reserved":0,"allocated":15732736,"data_stored":15110394,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.34699999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.318}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.254}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.505}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.57799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54672,"kb_used_data":15364,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317168,"statfs":{"total":96636764160,"available":96580780032,"internally_reserved":0,"allocated":15732736,"data_stored":15110394,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.311}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.64200000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.73499999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.32200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.25600000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67700000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.621},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.70999999999999996}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.73699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.495},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.67100000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509490,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38324,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39243776,"data_stored":40638270,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":1.149},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":1.2629999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":1.29},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.1899999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":1.327},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":1.556}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":1.4390000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.238}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":1.179},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":1.5309999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.3029999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.66100000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538448,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48712,"kb_used_data":20792,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323128,"statfs":{"total":96636764160,"available":96586883072,"internally_reserved":0,"allocated":21291008,"data_stored":21121998,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.35799999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.502}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61299999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.623},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:20.832 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:54:20.832 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 658, 'reported_epoch': 227, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:12.177819+0000', 'last_change': '2024-04-20T02:54:12.177819+0000', 'last_active': '2024-04-20T02:54:12.177819+0000', 'last_peered': '2024-04-20T02:54:12.177819+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:12.177819+0000', 'last_undegraded': '2024-04-20T02:54:12.177819+0000', 'last_fullsized': '2024-04-20T02:54:12.177819+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:54:20.832 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:54:21.234 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:21.234 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":253,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":228,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497258496,"bytes_avail":772596854784,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:54:21.249 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:54:21.249 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:21.573 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:21.574 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:21.586 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-04-20T02:54:21.491421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485632,"kb_used_data":209568,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489088,"statfs":{"total":773094113280,"available":772596826112,"internally_reserved":0,"allocated":214597632,"data_stored":217751662,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.325637"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509495,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.59899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.33900000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.50700000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70372,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301468,"statfs":{"total":96636764160,"available":96564703232,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.76200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.80400000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.56899999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.71299999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.624}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.871},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.72799999999999998}]}]},{"osd":2,"up_from":138,"seq":592705486871,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.253},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.34799999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.40500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.53100000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.45300000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.441},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.51800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54672,"kb_used_data":15364,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317168,"statfs":{"total":96636764160,"available":96580780032,"internally_reserved":0,"allocated":15732736,"data_stored":15110394,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.311}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.64200000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.73499999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.32200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.32300000000000001,"15min":0.33500000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.47699999999999998,"5min":0.751,"15min":0.751},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.37,"15min":0.378},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.83899999999999997,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.25600000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67700000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.621},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.68600000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.55300000000000005},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.083,"5min":1.153,"15min":1.153},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.34399999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.70999999999999996}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.73699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.495},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.67100000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509491,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.28699999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.503},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.53900000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.52900000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.55900000000000005}]}]},{"osd":4,"up_from":167,"seq":717259538449,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.623},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.44400000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.70999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.57599999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53700000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:24.590 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:24.924 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:24.926 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:24.937 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-04-20T02:54:23.492024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002718"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509495,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.59899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.33900000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.50700000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.625},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64900000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.503},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.61099999999999999}]}]},{"osd":2,"up_from":138,"seq":592705486871,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.253},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.34799999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.40500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.53100000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.45300000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.441},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.51800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.22800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.33000000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.57899999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.32900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.312},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.58499999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60699999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.70999999999999996}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.73699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.495},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.67100000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509491,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.28699999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.503},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.53900000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.52900000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.55900000000000005}]}]},{"osd":4,"up_from":167,"seq":717259538449,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.623},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.44400000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.70999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.57599999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53700000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:24.939 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:25.265 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:25.267 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:25.279 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-04-20T02:54:23.492024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002718"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509495,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.59899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.33900000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.50700000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.625},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64900000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.503},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.61099999999999999}]}]},{"osd":2,"up_from":138,"seq":592705486871,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.253},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.36399999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.34799999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.40500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.53100000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.45300000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.441},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.51800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.22800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.33000000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.57899999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.32900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.312},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.58499999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60699999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.70999999999999996}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.73699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.495},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.67100000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509491,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.28699999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.503},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.53900000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.52900000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.55900000000000005}]}]},{"osd":4,"up_from":167,"seq":717259538449,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.623},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.44400000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.70999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.57599999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53700000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:25.280 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:54:25.281 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 659, 'reported_epoch': 227, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:12.177923+0000', 'last_change': '2024-04-20T02:54:12.177819+0000', 'last_active': '2024-04-20T02:54:12.177923+0000', 'last_peered': '2024-04-20T02:54:12.177923+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:12.177923+0000', 'last_undegraded': '2024-04-20T02:54:12.177923+0000', 'last_fullsized': '2024-04-20T02:54:12.177923+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:54:25.281 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:54:25.692 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:25.693 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":258,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":228,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497295360,"bytes_avail":772596817920,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:54:25.707 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:54:25.707 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:26.032 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:26.034 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:26.046 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-04-20T02:54:25.492471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002666"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509496,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.441}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.36899999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.503},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54700000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.625},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64900000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.503},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.61099999999999999}]}]},{"osd":2,"up_from":138,"seq":592705486872,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.33400000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.311}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.28999999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.46600000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.41699999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.45500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.22800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.33000000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.57899999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.32900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.312},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.58499999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60699999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.70999999999999996}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.73699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.495},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.67100000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509492,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.39400000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.44},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.32800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.378},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.62},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.72999999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.74399999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.71799999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.70299999999999996}]}]},{"osd":4,"up_from":167,"seq":717259538450,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.442},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.499},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:29.048 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:29.387 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:29.389 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:29.400 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-04-20T02:54:27.493116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.339493"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509496,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.441}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.36899999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.503},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54700000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.625},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64900000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.503},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.61099999999999999}]}]},{"osd":2,"up_from":138,"seq":592705486872,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.33400000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.311}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.28999999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.46600000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.41699999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.45500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"min":{"1min":0.17999999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.375,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.33600000000000002,"15min":0.34599999999999997},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.22800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.39100000000000001,"15min":0.41199999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39400000000000002,"15min":0.39800000000000002},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.0609999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.33000000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.57899999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51400000000000001,"15min":0.52600000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70299999999999996,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61099999999999999,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.61899999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.70799999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.30399999999999999},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38600000000000001,"15min":0.38400000000000001},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.32900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.312},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.40400000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.58499999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70699999999999996,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.755,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60699999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.70999999999999996}]},{"osd":1,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.47699999999999998,"15min":0.48099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.58399999999999996,"5min":1.419,"15min":1.419},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.55000000000000004,"15min":0.58599999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59299999999999997,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.73699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:53:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.63200000000000001,"15min":0.71399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.63100000000000001,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.72299999999999998,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.61399999999999999,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.495},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.67100000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509492,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.39400000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.44},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.32800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.378},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.62},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.72999999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.74399999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.71799999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.70299999999999996}]}]},{"osd":4,"up_from":167,"seq":717259538450,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.442},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.499},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:29.402 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:29.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-04-20T02:54:29.728+0000 7f3aa37fe700 0 --2- 172.21.15.105:0/1669582885 >> [v2:172.21.15.105:3301/0,v1:172.21.15.105:6790/0] conn(0x7f3aa413ca40 0x7f3aa4131db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-20T02:54:29.929 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:29.931 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:29.942 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-04-20T02:54:29.493474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509496,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.441}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.36899999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.503},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54700000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.49099999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.40000000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.38100000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51300000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.55700000000000005}]}]},{"osd":2,"up_from":138,"seq":592705486872,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.33400000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.311}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.28999999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.46600000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.41699999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.45500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.247},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.30099999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.316}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.441},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.51000000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.46800000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.41799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.29099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28100000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.25900000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.52800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51800000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.501},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.54200000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.434}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.504}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.49099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.307},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.36199999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.38300000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.40300000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509492,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.39400000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.44},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.32800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.378},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.62},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.72999999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.74399999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.71799999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.70299999999999996}]}]},{"osd":4,"up_from":167,"seq":717259538450,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.442},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.499},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:29.944 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:54:29.944 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 659, 'reported_epoch': 227, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:12.177923+0000', 'last_change': '2024-04-20T02:54:12.177819+0000', 'last_active': '2024-04-20T02:54:12.177923+0000', 'last_peered': '2024-04-20T02:54:12.177923+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:12.177923+0000', 'last_undegraded': '2024-04-20T02:54:12.177923+0000', 'last_fullsized': '2024-04-20T02:54:12.177923+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:54:29.945 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:54:30.352 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:30.353 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":263,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":228,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497295360,"bytes_avail":772596817920,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:54:30.367 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:54:30.367 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:30.695 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:30.697 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:30.708 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-04-20T02:54:29.493474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509496,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.438,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.60399999999999998,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.52100000000000002,"15min":0.47399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52100000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.33300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.47999999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.60099999999999998,"15min":0.55800000000000005},"min":{"1min":0.313,"5min":0.307,"15min":0.307},"max":{"1min":1.3340000000000001,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54300000000000004,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.441}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.36899999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53500000000000003,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.503},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.55200000000000005,"15min":0.58499999999999996},"min":{"1min":0.35799999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.58699999999999997,"5min":1.599,"15min":1.599},"last":0.54700000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.49099999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.53200000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.40000000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.38100000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51300000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.55700000000000005}]}]},{"osd":2,"up_from":138,"seq":592705486872,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.33400000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.311}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.28999999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.46600000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.41699999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.45500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.247},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.30099999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.316}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.441},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.51000000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.46800000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.41799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.29099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.28100000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.25900000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.52800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51800000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.501},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.54200000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.68899999999999995,"15min":0.81499999999999995},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.55800000000000005,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.46200000000000002,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.61099999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.434}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.504}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.49099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.307},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.36199999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.38300000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.49399999999999999,"15min":0.47399999999999998},"min":{"1min":0.38400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.73799999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.57299999999999995,"15min":0.61499999999999999},"min":{"1min":0.379,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66700000000000004,"5min":3.98,"15min":3.98},"last":0.40300000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509492,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.219,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.98799999999999999,"5min":1.427,"15min":1.427},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.371,"15min":0.38},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":1.0049999999999999,"5min":1.427,"15min":1.427},"last":0.39400000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.44},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.32800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.42699999999999999,"15min":0.41199999999999998},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.1419999999999999,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.378},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0229999999999999,"5min":1.728,"15min":1.728},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.62},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.72999999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.74399999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56899999999999995,"15min":0.58799999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.3200000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58399999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.71799999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65200000000000002,"15min":0.69699999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":1.339,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.70299999999999996}]}]},{"osd":4,"up_from":167,"seq":717259538450,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.442},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.499},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:33.711 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:34.050 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:34.051 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:34.063 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-04-20T02:54:33.494430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003009"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509497,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.63600000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.625},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.61099999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.57399999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.748},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.76500000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.55900000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.436}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49399999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69499999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486873,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.252},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.35199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.313}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.39100000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.375},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.55000000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.437}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.246},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.224}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.29299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.30499999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.44600000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53500000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.51700000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54500000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.313},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.53000000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51500000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.622},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.47099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.377}]}]},{"osd":0,"up_from":15,"seq":64424509493,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.29399999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.35399999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.39300000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.51700000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.53300000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.62}]}]},{"osd":4,"up_from":167,"seq":717259538451,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.47799999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.433},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.497}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.52200000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.65100000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:34.065 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:34.388 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:34.390 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:34.402 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-04-20T02:54:33.494430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003009"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509497,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.63600000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.625},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.61099999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.57399999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.748},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.76500000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.55900000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.436}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49399999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69499999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486873,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.252},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.35199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.313}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.39100000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.375},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.55000000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.437}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.246},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.224}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.29299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.30499999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.44600000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53500000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.51700000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54500000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.313},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.53000000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51500000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.622},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.47099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.377}]}]},{"osd":0,"up_from":15,"seq":64424509493,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.29399999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.35399999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.39300000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.51700000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.53300000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.62}]}]},{"osd":4,"up_from":167,"seq":717259538451,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.47799999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.433},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.497}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.52200000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.65100000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:34.404 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:54:34.404 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 659, 'reported_epoch': 227, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:12.177923+0000', 'last_change': '2024-04-20T02:54:12.177819+0000', 'last_active': '2024-04-20T02:54:12.177923+0000', 'last_peered': '2024-04-20T02:54:12.177923+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:12.177923+0000', 'last_undegraded': '2024-04-20T02:54:12.177923+0000', 'last_fullsized': '2024-04-20T02:54:12.177923+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:54:34.404 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:54:34.808 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:34.808 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":267,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":228,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497295360,"bytes_avail":772596817920,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:54:34.823 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:54:34.823 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:35.148 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:35.150 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:35.161 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-04-20T02:54:33.494430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003009"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509497,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.63600000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.625},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.61099999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.57399999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.748},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.68500000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.76500000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.55900000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.436}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49399999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69499999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486873,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.252},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.35199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.313}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.39100000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.375},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.55000000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.437}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.246},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.224}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.29299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.30499999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.44600000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53500000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.51700000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54500000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.313},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.53000000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51500000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.622},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.47099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.377}]}]},{"osd":0,"up_from":15,"seq":64424509493,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.29399999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.35399999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.39300000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.51700000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.53300000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.62}]}]},{"osd":4,"up_from":167,"seq":717259538451,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.47799999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.433},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.497}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.52200000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.65100000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:38.164 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:38.506 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:38.507 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:38.518 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-04-20T02:54:37.495379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002907"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509498,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.627}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.60899999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.379}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.44700000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.55900000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.436}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49399999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69499999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.32500000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.307},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.28899999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.49399999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.48699999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.45600000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.52000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.246},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.224}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.29299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.30499999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.44600000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53500000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.51700000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54500000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.313},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.53000000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51500000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.622},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.47099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.377}]}]},{"osd":0,"up_from":15,"seq":64424509494,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.25},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.23599999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.31},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.502}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.60699999999999998}]}]},{"osd":4,"up_from":167,"seq":717259538452,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57799999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.442}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.5}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.40999999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:38.521 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:38.850 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:38.852 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:38.863 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-04-20T02:54:37.495379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485640,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489080,"statfs":{"total":773094113280,"available":772596817920,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002907"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":666,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169531+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:12.169531+0000","last_peered":"2024-04-20T02:54:12.169531+0000","last_clean":"2024-04-20T02:54:12.169531+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:12.169531+0000","last_undegraded":"2024-04-20T02:54:12.169531+0000","last_fullsized":"2024-04-20T02:54:12.169531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509498,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.627}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.60899999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.379}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.44700000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.55900000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.436}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49399999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.63500000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.67500000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69499999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.32500000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.307},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.28899999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.49399999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.48699999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.45600000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.52000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.246},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.224}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.29299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.30499999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.44600000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53500000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.51700000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.54500000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.313},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.53000000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51500000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83452,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288388,"statfs":{"total":96636764160,"available":96551309312,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.53800000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.622},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.56000000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.47099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.377}]}]},{"osd":0,"up_from":15,"seq":64424509494,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.25},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.23599999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.31},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.502}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.60699999999999998}]}]},{"osd":4,"up_from":167,"seq":717259538452,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57799999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.442}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.5}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.40999999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:38.865 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:54:38.866 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 659, 'reported_epoch': 227, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:12.177923+0000', 'last_change': '2024-04-20T02:54:12.177819+0000', 'last_active': '2024-04-20T02:54:12.177923+0000', 'last_peered': '2024-04-20T02:54:12.177923+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:12.177923+0000', 'last_undegraded': '2024-04-20T02:54:12.177923+0000', 'last_fullsized': '2024-04-20T02:54:12.177923+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:54:38.866 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:54:39.269 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:39.269 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":271,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":228,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497295360,"bytes_avail":772596817920,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:54:39.287 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:54:39.288 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:39.614 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:39.616 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:39.627 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-04-20T02:54:39.495800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485636,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489084,"statfs":{"total":773094113280,"available":772596822016,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002684"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.775218+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:12.775218+0000","last_peered":"2024-04-20T02:54:12.775218+0000","last_clean":"2024-04-20T02:54:12.775218+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:12.775218+0000","last_undegraded":"2024-04-20T02:54:12.775218+0000","last_fullsized":"2024-04-20T02:54:12.775218+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012742+0000","last_change":"2024-04-20T02:53:04.954637+0000","last_active":"2024-04-20T02:54:17.012742+0000","last_peered":"2024-04-20T02:54:17.012742+0000","last_clean":"2024-04-20T02:54:17.012742+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:17.012742+0000","last_undegraded":"2024-04-20T02:54:17.012742+0000","last_fullsized":"2024-04-20T02:54:17.012742+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-04-20T02:53:04.954524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:17.569030+0000","last_clean_scrub_stamp":"2024-04-20T02:53:04.954524+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:34.069583+0000","scrub_duration":0.0017228740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":667,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:16.895149+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:16.895149+0000","last_peered":"2024-04-20T02:54:16.895149+0000","last_clean":"2024-04-20T02:54:16.895149+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:16.895149+0000","last_undegraded":"2024-04-20T02:54:16.895149+0000","last_fullsized":"2024-04-20T02:54:16.895149+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":668,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.166985+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:12.166985+0000","last_peered":"2024-04-20T02:54:12.166985+0000","last_clean":"2024-04-20T02:54:12.166985+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:12.166985+0000","last_undegraded":"2024-04-20T02:54:12.166985+0000","last_fullsized":"2024-04-20T02:54:12.166985+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.002647+0000","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":243,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:16.895168+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:16.895168+0000","last_peered":"2024-04-20T02:54:16.895168+0000","last_clean":"2024-04-20T02:54:16.895168+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:16.895168+0000","last_undegraded":"2024-04-20T02:54:16.895168+0000","last_fullsized":"2024-04-20T02:54:16.895168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509498,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37748,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334092,"statfs":{"total":96636764160,"available":96598110208,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.627}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.60899999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.379}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.46100000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.44700000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.504},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.44500000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.47899999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.52800000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46100000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.53900000000000003}]}]},{"osd":2,"up_from":138,"seq":592705486874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43356,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328484,"statfs":{"total":96636764160,"available":96592367616,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.32500000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":4.609,"5min":4.609,"15min":4.609},"last":0.307},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.28899999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.49399999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.48699999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.45600000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":9.1419999999999995,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.52000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.437}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36299999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.52800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.504},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.57099999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.51800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574905,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.45200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34200000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.503},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.62}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.60899999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.497}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83448,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288392,"statfs":{"total":96636764160,"available":96551313408,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.73999999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.63},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.71699999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.70099999999999996}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.64300000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509494,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77644,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294196,"statfs":{"total":96636764160,"available":96557256704,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.25},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.23599999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.31},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.502}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.60699999999999998}]}]},{"osd":4,"up_from":167,"seq":717259538452,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57799999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.442}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.5}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.40999999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:42.630 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:42.970 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:42.972 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:42.983 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-04-20T02:54:41.496277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485624,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489096,"statfs":{"total":773094113280,"available":772596834304,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001846"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":271,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012744+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:17.012744+0000","last_peered":"2024-04-20T02:54:17.012744+0000","last_clean":"2024-04-20T02:54:17.012744+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:17.012744+0000","last_undegraded":"2024-04-20T02:54:17.012744+0000","last_fullsized":"2024-04-20T02:54:17.012744+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":407,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:34.852120+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:34.852120+0000","last_peered":"2024-04-20T02:54:34.852120+0000","last_clean":"2024-04-20T02:54:34.852120+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:34.852120+0000","last_undegraded":"2024-04-20T02:54:34.852120+0000","last_fullsized":"2024-04-20T02:54:34.852120+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":667,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:16.895149+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:16.895149+0000","last_peered":"2024-04-20T02:54:16.895149+0000","last_clean":"2024-04-20T02:54:16.895149+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:16.895149+0000","last_undegraded":"2024-04-20T02:54:16.895149+0000","last_fullsized":"2024-04-20T02:54:16.895149+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":669,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:16.155156+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:16.155156+0000","last_peered":"2024-04-20T02:54:16.155156+0000","last_clean":"2024-04-20T02:54:16.155156+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:16.155156+0000","last_undegraded":"2024-04-20T02:54:16.155156+0000","last_fullsized":"2024-04-20T02:54:16.155156+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":243,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:16.895168+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:16.895168+0000","last_peered":"2024-04-20T02:54:16.895168+0000","last_clean":"2024-04-20T02:54:16.895168+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:16.895168+0000","last_undegraded":"2024-04-20T02:54:16.895168+0000","last_fullsized":"2024-04-20T02:54:16.895168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37744,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334096,"statfs":{"total":96636764160,"available":96598114304,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.71299999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.746}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.503},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.58099999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.504},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.44500000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.47899999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.52800000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46100000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.53900000000000003}]}]},{"osd":2,"up_from":138,"seq":592705486875,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43352,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328488,"statfs":{"total":96636764160,"available":96592371712,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.33500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.58099999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.496}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.437}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36299999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.52800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.504},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.57099999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.51800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574905,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.45200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34200000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.503},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.62}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.60899999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.497}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83448,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288392,"statfs":{"total":96636764160,"available":96551313408,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.73999999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.63},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.71699999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.70099999999999996}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.64300000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509495,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77640,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294200,"statfs":{"total":96636764160,"available":96557260800,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.25},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.23599999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.31},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.502}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.60699999999999998}]}]},{"osd":4,"up_from":167,"seq":717259538453,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.58599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.67200000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.71799999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.754},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.74399999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.81399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:42.986 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:43.315 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:43.317 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:43.328 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-04-20T02:54:41.496277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485624,"kb_used_data":209580,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754489096,"statfs":{"total":773094113280,"available":772596834304,"internally_reserved":0,"allocated":214609920,"data_stored":217756426,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001846"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":191,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.082000+0000","last_change":"2024-04-20T02:53:56.572693+0000","last_active":"2024-04-20T02:54:13.082000+0000","last_peered":"2024-04-20T02:54:13.082000+0000","last_clean":"2024-04-20T02:54:13.082000+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:13.082000+0000","last_undegraded":"2024-04-20T02:54:13.082000+0000","last_fullsized":"2024-04-20T02:54:13.082000+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:30.912740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:30.912740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:38.061478+0000","scrub_duration":0.001422855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":260,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.169643+0000","last_change":"2024-04-20T02:53:55.393174+0000","last_active":"2024-04-20T02:54:12.169643+0000","last_peered":"2024-04-20T02:54:12.169643+0000","last_clean":"2024-04-20T02:54:12.169643+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:12.169643+0000","last_undegraded":"2024-04-20T02:54:12.169643+0000","last_fullsized":"2024-04-20T02:54:12.169643+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:19.107014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:19.107014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:40.198494+0000","scrub_duration":0.0018833859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":271,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012744+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:17.012744+0000","last_peered":"2024-04-20T02:54:17.012744+0000","last_clean":"2024-04-20T02:54:17.012744+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:17.012744+0000","last_undegraded":"2024-04-20T02:54:17.012744+0000","last_fullsized":"2024-04-20T02:54:17.012744+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:13.081985+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:13.081985+0000","last_peered":"2024-04-20T02:54:13.081985+0000","last_clean":"2024-04-20T02:54:13.081985+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:13.081985+0000","last_undegraded":"2024-04-20T02:54:13.081985+0000","last_fullsized":"2024-04-20T02:54:13.081985+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":643,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.170767+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:12.170767+0000","last_peered":"2024-04-20T02:54:12.170767+0000","last_clean":"2024-04-20T02:54:12.170767+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:12.170767+0000","last_undegraded":"2024-04-20T02:54:12.170767+0000","last_fullsized":"2024-04-20T02:54:12.170767+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":407,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:34.852120+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:34.852120+0000","last_peered":"2024-04-20T02:54:34.852120+0000","last_clean":"2024-04-20T02:54:34.852120+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:34.852120+0000","last_undegraded":"2024-04-20T02:54:34.852120+0000","last_fullsized":"2024-04-20T02:54:34.852120+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":248,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.550328+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:17.550328+0000","last_peered":"2024-04-20T02:54:17.550328+0000","last_clean":"2024-04-20T02:54:17.550328+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:17.550328+0000","last_undegraded":"2024-04-20T02:54:17.550328+0000","last_fullsized":"2024-04-20T02:54:17.550328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":667,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:16.895149+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:16.895149+0000","last_peered":"2024-04-20T02:54:16.895149+0000","last_clean":"2024-04-20T02:54:16.895149+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:16.895149+0000","last_undegraded":"2024-04-20T02:54:16.895149+0000","last_fullsized":"2024-04-20T02:54:16.895149+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":669,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:16.155156+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:16.155156+0000","last_peered":"2024-04-20T02:54:16.155156+0000","last_clean":"2024-04-20T02:54:16.155156+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:16.155156+0000","last_undegraded":"2024-04-20T02:54:16.155156+0000","last_fullsized":"2024-04-20T02:54:16.155156+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.173358+0000","last_change":"2024-04-20T02:54:08.829686+0000","last_active":"2024-04-20T02:54:12.173358+0000","last_peered":"2024-04-20T02:54:12.173358+0000","last_clean":"2024-04-20T02:54:12.173358+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:12.173358+0000","last_undegraded":"2024-04-20T02:54:12.173358+0000","last_fullsized":"2024-04-20T02:54:12.173358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:32.389879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:32.389879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:36.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":243,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:16.895168+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:16.895168+0000","last_peered":"2024-04-20T02:54:16.895168+0000","last_clean":"2024-04-20T02:54:16.895168+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:16.895168+0000","last_undegraded":"2024-04-20T02:54:16.895168+0000","last_fullsized":"2024-04-20T02:54:16.895168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37744,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334096,"statfs":{"total":96636764160,"available":96598114304,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.71299999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.746}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.503},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.58099999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.504},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.44500000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":9.7829999999999995,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.47899999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.52800000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46100000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.53900000000000003}]}]},{"osd":2,"up_from":138,"seq":592705486875,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43352,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328488,"statfs":{"total":96636764160,"available":96592371712,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.33500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.58099999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.496}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54684,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317156,"statfs":{"total":96636764160,"available":96580767744,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.437}]},{"osd":2,"last update":"Sat Apr 20 02:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36299999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.52800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.504},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.57099999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.51800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574905,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69668,"kb_used_data":34456,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302172,"statfs":{"total":96636764160,"available":96565424128,"internally_reserved":0,"allocated":35282944,"data_stored":36196616,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.45200000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34200000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:53:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.503},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.62}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.60899999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.497}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51500000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83448,"kb_used_data":48240,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288392,"statfs":{"total":96636764160,"available":96551313408,"internally_reserved":0,"allocated":49397760,"data_stored":51688188,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.73999999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.63},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.71699999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.70099999999999996}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.64300000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509495,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77640,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294200,"statfs":{"total":96636764160,"available":96557260800,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.25},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.23599999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.31},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.502}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.60699999999999998}]}]},{"osd":4,"up_from":167,"seq":717259538453,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.58599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.67200000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.71799999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.754},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.74399999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.81399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:43.330 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:54:43.330 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 659, 'reported_epoch': 227, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:12.177923+0000', 'last_change': '2024-04-20T02:54:12.177819+0000', 'last_active': '2024-04-20T02:54:12.177923+0000', 'last_peered': '2024-04-20T02:54:12.177923+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:12.177923+0000', 'last_undegraded': '2024-04-20T02:54:12.177923+0000', 'last_fullsized': '2024-04-20T02:54:12.177923+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:54:43.330 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:54:43.735 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:43.735 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":276,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":230,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497278976,"bytes_avail":772596834304,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:54:43.749 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:54:43.749 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:44.076 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:44.078 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:44.089 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-04-20T02:54:43.496697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485724,"kb_used_data":209664,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488996,"statfs":{"total":773094113280,"available":772596731904,"internally_reserved":0,"allocated":214695936,"data_stored":217789974,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001819"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530769+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:40.530769+0000","last_peered":"2024-04-20T02:54:40.530769+0000","last_clean":"2024-04-20T02:54:40.530769+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:40.530769+0000","last_undegraded":"2024-04-20T02:54:40.530769+0000","last_fullsized":"2024-04-20T02:54:40.530769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.861955+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:41.861955+0000","last_peered":"2024-04-20T02:54:41.861955+0000","last_clean":"2024-04-20T02:54:41.861955+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:41.861955+0000","last_undegraded":"2024-04-20T02:54:41.861955+0000","last_fullsized":"2024-04-20T02:54:41.861955+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":271,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:17.012744+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:17.012744+0000","last_peered":"2024-04-20T02:54:17.012744+0000","last_clean":"2024-04-20T02:54:17.012744+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:17.012744+0000","last_undegraded":"2024-04-20T02:54:17.012744+0000","last_fullsized":"2024-04-20T02:54:17.012744+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":659,"reported_epoch":227,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:12.177923+0000","last_change":"2024-04-20T02:54:12.177819+0000","last_active":"2024-04-20T02:54:12.177923+0000","last_peered":"2024-04-20T02:54:12.177923+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:12.177923+0000","last_undegraded":"2024-04-20T02:54:12.177923+0000","last_fullsized":"2024-04-20T02:54:12.177923+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:28.272460+0000","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530766+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:40.530766+0000","last_peered":"2024-04-20T02:54:40.530766+0000","last_clean":"2024-04-20T02:54:40.530766+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:40.530766+0000","last_undegraded":"2024-04-20T02:54:40.530766+0000","last_fullsized":"2024-04-20T02:54:40.530766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":645,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178322+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:40.178322+0000","last_peered":"2024-04-20T02:54:40.178322+0000","last_clean":"2024-04-20T02:54:40.178322+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:40.178322+0000","last_undegraded":"2024-04-20T02:54:40.178322+0000","last_fullsized":"2024-04-20T02:54:40.178322+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":407,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:34.852120+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:34.852120+0000","last_peered":"2024-04-20T02:54:34.852120+0000","last_clean":"2024-04-20T02:54:34.852120+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:34.852120+0000","last_undegraded":"2024-04-20T02:54:34.852120+0000","last_fullsized":"2024-04-20T02:54:34.852120+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531139+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:40.531139+0000","last_peered":"2024-04-20T02:54:40.531139+0000","last_clean":"2024-04-20T02:54:40.531139+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:40.531139+0000","last_undegraded":"2024-04-20T02:54:40.531139+0000","last_fullsized":"2024-04-20T02:54:40.531139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":669,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178166+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:40.178166+0000","last_peered":"2024-04-20T02:54:40.178166+0000","last_clean":"2024-04-20T02:54:40.178166+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:40.178166+0000","last_undegraded":"2024-04-20T02:54:40.178166+0000","last_fullsized":"2024-04-20T02:54:40.178166+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-04-20T02:54:12.795718+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:12.795718+0000","last_peered":"2024-04-20T02:54:12.795718+0000","last_clean":"2024-04-20T02:54:12.795718+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:12.795718+0000","last_undegraded":"2024-04-20T02:54:12.795718+0000","last_fullsized":"2024-04-20T02:54:12.795718+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":669,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-04-20T02:54:16.155156+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:16.155156+0000","last_peered":"2024-04-20T02:54:16.155156+0000","last_clean":"2024-04-20T02:54:16.155156+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:16.155156+0000","last_undegraded":"2024-04-20T02:54:16.155156+0000","last_fullsized":"2024-04-20T02:54:16.155156+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530964+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:40.530964+0000","last_peered":"2024-04-20T02:54:40.530964+0000","last_clean":"2024-04-20T02:54:40.530964+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:40.530964+0000","last_undegraded":"2024-04-20T02:54:40.530964+0000","last_fullsized":"2024-04-20T02:54:40.530964+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178162+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:40.178162+0000","last_peered":"2024-04-20T02:54:40.178162+0000","last_clean":"2024-04-20T02:54:40.178162+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:40.178162+0000","last_undegraded":"2024-04-20T02:54:40.178162+0000","last_fullsized":"2024-04-20T02:54:40.178162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37744,"kb_used_data":2536,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334096,"statfs":{"total":96636764160,"available":96598114304,"internally_reserved":0,"allocated":2596864,"data_stored":1674894,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.71299999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.746}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.503},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.58099999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70400,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301440,"statfs":{"total":96636764160,"available":96564674560,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51400000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.54000000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.55000000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.316},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.38800000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.34999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.495}]}]},{"osd":2,"up_from":138,"seq":592705486875,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43352,"kb_used_data":15376,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328488,"statfs":{"total":96636764160,"available":96592371712,"internally_reserved":0,"allocated":15745024,"data_stored":15115158,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.33500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.58099999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.496}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.437}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.36599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.436}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":1.0149999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69680,"kb_used_data":34468,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302160,"statfs":{"total":96636764160,"available":96565411840,"internally_reserved":0,"allocated":35295232,"data_stored":36201430,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.43099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.66200000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68400000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83476,"kb_used_data":48264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288364,"statfs":{"total":96636764160,"available":96551284736,"internally_reserved":0,"allocated":49422336,"data_stored":51697766,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.753},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.70799999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.73499999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72399999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.69099999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.46000000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.67200000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509495,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77640,"kb_used_data":38336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294200,"statfs":{"total":96636764160,"available":96557260800,"internally_reserved":0,"allocated":39256064,"data_stored":40643034,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.25},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.23599999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.28100000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.31},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.502}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.60699999999999998}]}]},{"osd":4,"up_from":167,"seq":717259538453,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":20804,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":21303296,"data_stored":21126762,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.58599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.67200000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.70299999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.71799999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.754},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.74399999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.81399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:47.092 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:47.445 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:47.447 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:47.458 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-04-20T02:54:45.497106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485840,"kb_used_data":209760,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488880,"statfs":{"total":773094113280,"available":772596613120,"internally_reserved":0,"allocated":214794240,"data_stored":217828286,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001727"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530769+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:40.530769+0000","last_peered":"2024-04-20T02:54:40.530769+0000","last_clean":"2024-04-20T02:54:40.530769+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:40.530769+0000","last_undegraded":"2024-04-20T02:54:40.530769+0000","last_fullsized":"2024-04-20T02:54:40.530769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.861955+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:41.861955+0000","last_peered":"2024-04-20T02:54:41.861955+0000","last_clean":"2024-04-20T02:54:41.861955+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:41.861955+0000","last_undegraded":"2024-04-20T02:54:41.861955+0000","last_fullsized":"2024-04-20T02:54:41.861955+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531047+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:40.531047+0000","last_peered":"2024-04-20T02:54:40.531047+0000","last_clean":"2024-04-20T02:54:40.531047+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:40.531047+0000","last_undegraded":"2024-04-20T02:54:40.531047+0000","last_fullsized":"2024-04-20T02:54:40.531047+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":666,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:42.178920+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:42.178920+0000","last_peered":"2024-04-20T02:54:42.178920+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:42.178920+0000","last_undegraded":"2024-04-20T02:54:42.178920+0000","last_fullsized":"2024-04-20T02:54:42.178920+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530766+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:40.530766+0000","last_peered":"2024-04-20T02:54:40.530766+0000","last_clean":"2024-04-20T02:54:40.530766+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:40.530766+0000","last_undegraded":"2024-04-20T02:54:40.530766+0000","last_fullsized":"2024-04-20T02:54:40.530766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":645,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178322+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:40.178322+0000","last_peered":"2024-04-20T02:54:40.178322+0000","last_clean":"2024-04-20T02:54:40.178322+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:40.178322+0000","last_undegraded":"2024-04-20T02:54:40.178322+0000","last_fullsized":"2024-04-20T02:54:40.178322+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":409,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530924+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:40.530924+0000","last_peered":"2024-04-20T02:54:40.530924+0000","last_clean":"2024-04-20T02:54:40.530924+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:40.530924+0000","last_undegraded":"2024-04-20T02:54:40.530924+0000","last_fullsized":"2024-04-20T02:54:40.530924+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531139+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:40.531139+0000","last_peered":"2024-04-20T02:54:40.531139+0000","last_clean":"2024-04-20T02:54:40.531139+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:40.531139+0000","last_undegraded":"2024-04-20T02:54:40.531139+0000","last_fullsized":"2024-04-20T02:54:40.531139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":669,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178166+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:40.178166+0000","last_peered":"2024-04-20T02:54:40.178166+0000","last_clean":"2024-04-20T02:54:40.178166+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:40.178166+0000","last_undegraded":"2024-04-20T02:54:40.178166+0000","last_fullsized":"2024-04-20T02:54:40.178166+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530242+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:40.530242+0000","last_peered":"2024-04-20T02:54:40.530242+0000","last_clean":"2024-04-20T02:54:40.530242+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:40.530242+0000","last_undegraded":"2024-04-20T02:54:40.530242+0000","last_fullsized":"2024-04-20T02:54:40.530242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":670,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178327+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:40.178327+0000","last_peered":"2024-04-20T02:54:40.178327+0000","last_clean":"2024-04-20T02:54:40.178327+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:40.178327+0000","last_undegraded":"2024-04-20T02:54:40.178327+0000","last_fullsized":"2024-04-20T02:54:40.178327+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530964+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:40.530964+0000","last_peered":"2024-04-20T02:54:40.530964+0000","last_clean":"2024-04-20T02:54:40.530964+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:40.530964+0000","last_undegraded":"2024-04-20T02:54:40.530964+0000","last_fullsized":"2024-04-20T02:54:40.530964+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178162+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:40.178162+0000","last_peered":"2024-04-20T02:54:40.178162+0000","last_clean":"2024-04-20T02:54:40.178162+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:40.178162+0000","last_undegraded":"2024-04-20T02:54:40.178162+0000","last_fullsized":"2024-04-20T02:54:40.178162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37776,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334064,"statfs":{"total":96636764160,"available":96598081536,"internally_reserved":0,"allocated":2621440,"data_stored":1684472,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.73599999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.65300000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.67400000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.54300000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.52800000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70400,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301440,"statfs":{"total":96636764160,"available":96564674560,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51400000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.54000000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.55000000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.316},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.38800000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.34999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.495}]}]},{"osd":2,"up_from":138,"seq":592705486876,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43380,"kb_used_data":15400,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328460,"statfs":{"total":96636764160,"available":96592343040,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.30499999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.254},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.314}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.51400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.52700000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.63300000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.57499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.437}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.36599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.436}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":1.0149999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69680,"kb_used_data":34468,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302160,"statfs":{"total":96636764160,"available":96565411840,"internally_reserved":0,"allocated":35295232,"data_stored":36201430,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.43099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.66200000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68400000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83476,"kb_used_data":48264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288364,"statfs":{"total":96636764160,"available":96551284736,"internally_reserved":0,"allocated":49422336,"data_stored":51697766,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.753},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.70799999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.73499999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72399999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.69099999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.46000000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.67200000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509496,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77672,"kb_used_data":38360,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294168,"statfs":{"total":96636764160,"available":96557228032,"internally_reserved":0,"allocated":39280640,"data_stored":40652612,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.37}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.439}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.68999999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.74399999999999999}]}]},{"osd":4,"up_from":167,"seq":717259538454,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48744,"kb_used_data":20828,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323096,"statfs":{"total":96636764160,"available":96586850304,"internally_reserved":0,"allocated":21327872,"data_stored":21136340,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.876},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.72599999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.5},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.82899999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.93799999999999994}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.84599999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:47.460 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:47.784 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:47.786 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:47.797 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-04-20T02:54:47.497494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485840,"kb_used_data":209760,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488880,"statfs":{"total":773094113280,"available":772596613120,"internally_reserved":0,"allocated":214794240,"data_stored":217828286,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001693"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530769+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:40.530769+0000","last_peered":"2024-04-20T02:54:40.530769+0000","last_clean":"2024-04-20T02:54:40.530769+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:40.530769+0000","last_undegraded":"2024-04-20T02:54:40.530769+0000","last_fullsized":"2024-04-20T02:54:40.530769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.861955+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:41.861955+0000","last_peered":"2024-04-20T02:54:41.861955+0000","last_clean":"2024-04-20T02:54:41.861955+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:41.861955+0000","last_undegraded":"2024-04-20T02:54:41.861955+0000","last_fullsized":"2024-04-20T02:54:41.861955+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531047+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:40.531047+0000","last_peered":"2024-04-20T02:54:40.531047+0000","last_clean":"2024-04-20T02:54:40.531047+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:40.531047+0000","last_undegraded":"2024-04-20T02:54:40.531047+0000","last_fullsized":"2024-04-20T02:54:40.531047+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":666,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:42.178920+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:42.178920+0000","last_peered":"2024-04-20T02:54:42.178920+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:42.178920+0000","last_undegraded":"2024-04-20T02:54:42.178920+0000","last_fullsized":"2024-04-20T02:54:42.178920+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530766+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:40.530766+0000","last_peered":"2024-04-20T02:54:40.530766+0000","last_clean":"2024-04-20T02:54:40.530766+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:40.530766+0000","last_undegraded":"2024-04-20T02:54:40.530766+0000","last_fullsized":"2024-04-20T02:54:40.530766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":645,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178322+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:40.178322+0000","last_peered":"2024-04-20T02:54:40.178322+0000","last_clean":"2024-04-20T02:54:40.178322+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:40.178322+0000","last_undegraded":"2024-04-20T02:54:40.178322+0000","last_fullsized":"2024-04-20T02:54:40.178322+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":409,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530924+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:40.530924+0000","last_peered":"2024-04-20T02:54:40.530924+0000","last_clean":"2024-04-20T02:54:40.530924+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:40.530924+0000","last_undegraded":"2024-04-20T02:54:40.530924+0000","last_fullsized":"2024-04-20T02:54:40.530924+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531139+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:40.531139+0000","last_peered":"2024-04-20T02:54:40.531139+0000","last_clean":"2024-04-20T02:54:40.531139+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:40.531139+0000","last_undegraded":"2024-04-20T02:54:40.531139+0000","last_fullsized":"2024-04-20T02:54:40.531139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":669,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178166+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:40.178166+0000","last_peered":"2024-04-20T02:54:40.178166+0000","last_clean":"2024-04-20T02:54:40.178166+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:40.178166+0000","last_undegraded":"2024-04-20T02:54:40.178166+0000","last_fullsized":"2024-04-20T02:54:40.178166+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530242+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:40.530242+0000","last_peered":"2024-04-20T02:54:40.530242+0000","last_clean":"2024-04-20T02:54:40.530242+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:40.530242+0000","last_undegraded":"2024-04-20T02:54:40.530242+0000","last_fullsized":"2024-04-20T02:54:40.530242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":670,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178327+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:40.178327+0000","last_peered":"2024-04-20T02:54:40.178327+0000","last_clean":"2024-04-20T02:54:40.178327+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:40.178327+0000","last_undegraded":"2024-04-20T02:54:40.178327+0000","last_fullsized":"2024-04-20T02:54:40.178327+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530964+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:40.530964+0000","last_peered":"2024-04-20T02:54:40.530964+0000","last_clean":"2024-04-20T02:54:40.530964+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:40.530964+0000","last_undegraded":"2024-04-20T02:54:40.530964+0000","last_fullsized":"2024-04-20T02:54:40.530964+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178162+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:40.178162+0000","last_peered":"2024-04-20T02:54:40.178162+0000","last_clean":"2024-04-20T02:54:40.178162+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:40.178162+0000","last_undegraded":"2024-04-20T02:54:40.178162+0000","last_fullsized":"2024-04-20T02:54:40.178162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37776,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334064,"statfs":{"total":96636764160,"available":96598081536,"internally_reserved":0,"allocated":2621440,"data_stored":1684472,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.73599999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.65300000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.67400000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.54300000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.52800000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70400,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301440,"statfs":{"total":96636764160,"available":96564674560,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51400000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.54000000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.55000000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.316},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.38800000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.34999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.495}]}]},{"osd":2,"up_from":138,"seq":592705486876,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43380,"kb_used_data":15400,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328460,"statfs":{"total":96636764160,"available":96592343040,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.30499999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.254},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.314}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.51400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.52700000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.63300000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.57499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.437}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.36599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.436}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":1.0149999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69680,"kb_used_data":34468,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302160,"statfs":{"total":96636764160,"available":96565411840,"internally_reserved":0,"allocated":35295232,"data_stored":36201430,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.43099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.66200000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68400000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83476,"kb_used_data":48264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288364,"statfs":{"total":96636764160,"available":96551284736,"internally_reserved":0,"allocated":49422336,"data_stored":51697766,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.753},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.70799999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.73499999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72399999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.69099999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.46000000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.67200000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509496,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77672,"kb_used_data":38360,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294168,"statfs":{"total":96636764160,"available":96557228032,"internally_reserved":0,"allocated":39280640,"data_stored":40652612,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.37}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.439}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.68999999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.74399999999999999}]}]},{"osd":4,"up_from":167,"seq":717259538454,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48744,"kb_used_data":20828,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323096,"statfs":{"total":96636764160,"available":96586850304,"internally_reserved":0,"allocated":21327872,"data_stored":21136340,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.876},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.72599999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.5},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.82899999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.93799999999999994}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.84599999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:47.799 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:54:47.799 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 666, 'reported_epoch': 230, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:42.178920+0000', 'last_change': '2024-04-20T02:54:42.178857+0000', 'last_active': '2024-04-20T02:54:42.178920+0000', 'last_peered': '2024-04-20T02:54:42.178920+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:42.178920+0000', 'last_undegraded': '2024-04-20T02:54:42.178920+0000', 'last_fullsized': '2024-04-20T02:54:42.178920+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:54:47.800 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:54:48.202 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:48.202 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":280,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":230,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497500160,"bytes_avail":772596613120,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:54:48.219 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:54:48.219 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:48.545 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:48.546 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:48.557 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-04-20T02:54:47.497494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485840,"kb_used_data":209760,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488880,"statfs":{"total":773094113280,"available":772596613120,"internally_reserved":0,"allocated":214794240,"data_stored":217828286,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001693"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530769+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:40.530769+0000","last_peered":"2024-04-20T02:54:40.530769+0000","last_clean":"2024-04-20T02:54:40.530769+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:40.530769+0000","last_undegraded":"2024-04-20T02:54:40.530769+0000","last_fullsized":"2024-04-20T02:54:40.530769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.861955+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:41.861955+0000","last_peered":"2024-04-20T02:54:41.861955+0000","last_clean":"2024-04-20T02:54:41.861955+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:41.861955+0000","last_undegraded":"2024-04-20T02:54:41.861955+0000","last_fullsized":"2024-04-20T02:54:41.861955+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531047+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:40.531047+0000","last_peered":"2024-04-20T02:54:40.531047+0000","last_clean":"2024-04-20T02:54:40.531047+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:40.531047+0000","last_undegraded":"2024-04-20T02:54:40.531047+0000","last_fullsized":"2024-04-20T02:54:40.531047+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":666,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:42.178920+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:42.178920+0000","last_peered":"2024-04-20T02:54:42.178920+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:42.178920+0000","last_undegraded":"2024-04-20T02:54:42.178920+0000","last_fullsized":"2024-04-20T02:54:42.178920+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530766+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:40.530766+0000","last_peered":"2024-04-20T02:54:40.530766+0000","last_clean":"2024-04-20T02:54:40.530766+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:40.530766+0000","last_undegraded":"2024-04-20T02:54:40.530766+0000","last_fullsized":"2024-04-20T02:54:40.530766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":645,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178322+0000","last_change":"2024-04-20T02:53:16.250068+0000","last_active":"2024-04-20T02:54:40.178322+0000","last_peered":"2024-04-20T02:54:40.178322+0000","last_clean":"2024-04-20T02:54:40.178322+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:40.178322+0000","last_undegraded":"2024-04-20T02:54:40.178322+0000","last_fullsized":"2024-04-20T02:54:40.178322+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'267","last_scrub_stamp":"2024-04-20T02:53:16.249931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:16.249931+0000","objects_scrubbed":32,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:54:43.965306+0000","scrub_duration":0.0056808600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":409,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530924+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:40.530924+0000","last_peered":"2024-04-20T02:54:40.530924+0000","last_clean":"2024-04-20T02:54:40.530924+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:40.530924+0000","last_undegraded":"2024-04-20T02:54:40.530924+0000","last_fullsized":"2024-04-20T02:54:40.530924+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531139+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:40.531139+0000","last_peered":"2024-04-20T02:54:40.531139+0000","last_clean":"2024-04-20T02:54:40.531139+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:40.531139+0000","last_undegraded":"2024-04-20T02:54:40.531139+0000","last_fullsized":"2024-04-20T02:54:40.531139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":669,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178166+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:40.178166+0000","last_peered":"2024-04-20T02:54:40.178166+0000","last_clean":"2024-04-20T02:54:40.178166+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:40.178166+0000","last_undegraded":"2024-04-20T02:54:40.178166+0000","last_fullsized":"2024-04-20T02:54:40.178166+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530242+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:40.530242+0000","last_peered":"2024-04-20T02:54:40.530242+0000","last_clean":"2024-04-20T02:54:40.530242+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:40.530242+0000","last_undegraded":"2024-04-20T02:54:40.530242+0000","last_fullsized":"2024-04-20T02:54:40.530242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":670,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178327+0000","last_change":"2024-04-20T02:53:58.158125+0000","last_active":"2024-04-20T02:54:40.178327+0000","last_peered":"2024-04-20T02:54:40.178327+0000","last_clean":"2024-04-20T02:54:40.178327+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:40.178327+0000","last_undegraded":"2024-04-20T02:54:40.178327+0000","last_fullsized":"2024-04-20T02:54:40.178327+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'232","last_scrub_stamp":"2024-04-20T02:53:18.737634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:18.737634+0000","objects_scrubbed":33,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076830199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530964+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:40.530964+0000","last_peered":"2024-04-20T02:54:40.530964+0000","last_clean":"2024-04-20T02:54:40.530964+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:40.530964+0000","last_undegraded":"2024-04-20T02:54:40.530964+0000","last_fullsized":"2024-04-20T02:54:40.530964+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178162+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:40.178162+0000","last_peered":"2024-04-20T02:54:40.178162+0000","last_clean":"2024-04-20T02:54:40.178162+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:40.178162+0000","last_undegraded":"2024-04-20T02:54:40.178162+0000","last_fullsized":"2024-04-20T02:54:40.178162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37776,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334064,"statfs":{"total":96636764160,"available":96598081536,"internally_reserved":0,"allocated":2621440,"data_stored":1684472,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.73599999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.65300000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.67400000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.54300000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.438},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.52800000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70400,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301440,"statfs":{"total":96636764160,"available":96564674560,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51400000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.54000000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.55000000000000004}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.316},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.38800000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.34999999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.495}]}]},{"osd":2,"up_from":138,"seq":592705486876,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43380,"kb_used_data":15400,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328460,"statfs":{"total":96636764160,"available":96592343040,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.30499999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.254},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.314}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.51400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.52700000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.63300000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.57499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.437}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.36599999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.436}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":1.0149999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69680,"kb_used_data":34468,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302160,"statfs":{"total":96636764160,"available":96565411840,"internally_reserved":0,"allocated":35295232,"data_stored":36201430,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.47399999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.43099999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.66200000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68400000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83476,"kb_used_data":48264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288364,"statfs":{"total":96636764160,"available":96551284736,"internally_reserved":0,"allocated":49422336,"data_stored":51697766,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.753},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.70799999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.73499999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.72399999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.69099999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.46000000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.67200000000000004}]}]},{"osd":0,"up_from":15,"seq":64424509496,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77672,"kb_used_data":38360,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294168,"statfs":{"total":96636764160,"available":96557228032,"internally_reserved":0,"allocated":39280640,"data_stored":40652612,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.37}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.439}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.68999999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.74399999999999999}]}]},{"osd":4,"up_from":167,"seq":717259538454,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48744,"kb_used_data":20828,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323096,"statfs":{"total":96636764160,"available":96586850304,"internally_reserved":0,"allocated":21327872,"data_stored":21136340,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.876},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.72599999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.5},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.82899999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.93799999999999994}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.84599999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:51.560 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:51.900 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:51.901 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:51.913 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-04-20T02:54:51.498218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485844,"kb_used_data":209772,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488876,"statfs":{"total":773094113280,"available":772596609024,"internally_reserved":0,"allocated":214806528,"data_stored":217833050,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.338183"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530769+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:40.530769+0000","last_peered":"2024-04-20T02:54:40.530769+0000","last_clean":"2024-04-20T02:54:40.530769+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:40.530769+0000","last_undegraded":"2024-04-20T02:54:40.530769+0000","last_fullsized":"2024-04-20T02:54:40.530769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":274,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.862079+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:41.862079+0000","last_peered":"2024-04-20T02:54:41.862079+0000","last_clean":"2024-04-20T02:54:41.862079+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:41.862079+0000","last_undegraded":"2024-04-20T02:54:41.862079+0000","last_fullsized":"2024-04-20T02:54:41.862079+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531047+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:40.531047+0000","last_peered":"2024-04-20T02:54:40.531047+0000","last_clean":"2024-04-20T02:54:40.531047+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:40.531047+0000","last_undegraded":"2024-04-20T02:54:40.531047+0000","last_fullsized":"2024-04-20T02:54:40.531047+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":666,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:42.178920+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:42.178920+0000","last_peered":"2024-04-20T02:54:42.178920+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:42.178920+0000","last_undegraded":"2024-04-20T02:54:42.178920+0000","last_fullsized":"2024-04-20T02:54:42.178920+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530766+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:40.530766+0000","last_peered":"2024-04-20T02:54:40.530766+0000","last_clean":"2024-04-20T02:54:40.530766+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:40.530766+0000","last_undegraded":"2024-04-20T02:54:40.530766+0000","last_fullsized":"2024-04-20T02:54:40.530766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":657,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:44.903595+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:44.903595+0000","last_peered":"2024-04-20T02:54:44.903595+0000","last_clean":"2024-04-20T02:54:44.903595+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:44.903595+0000","last_undegraded":"2024-04-20T02:54:44.903595+0000","last_fullsized":"2024-04-20T02:54:44.903595+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":409,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530924+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:40.530924+0000","last_peered":"2024-04-20T02:54:40.530924+0000","last_clean":"2024-04-20T02:54:40.530924+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:40.530924+0000","last_undegraded":"2024-04-20T02:54:40.530924+0000","last_fullsized":"2024-04-20T02:54:40.530924+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531139+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:40.531139+0000","last_peered":"2024-04-20T02:54:40.531139+0000","last_clean":"2024-04-20T02:54:40.531139+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:40.531139+0000","last_undegraded":"2024-04-20T02:54:40.531139+0000","last_fullsized":"2024-04-20T02:54:40.531139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":669,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178166+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:40.178166+0000","last_peered":"2024-04-20T02:54:40.178166+0000","last_clean":"2024-04-20T02:54:40.178166+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:40.178166+0000","last_undegraded":"2024-04-20T02:54:40.178166+0000","last_fullsized":"2024-04-20T02:54:40.178166+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530242+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:40.530242+0000","last_peered":"2024-04-20T02:54:40.530242+0000","last_clean":"2024-04-20T02:54:40.530242+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:40.530242+0000","last_undegraded":"2024-04-20T02:54:40.530242+0000","last_fullsized":"2024-04-20T02:54:40.530242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":681,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:46.470549+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:54:46.470549+0000","last_peered":"2024-04-20T02:54:46.470549+0000","last_clean":"2024-04-20T02:54:46.470549+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:46.470549+0000","last_undegraded":"2024-04-20T02:54:46.470549+0000","last_fullsized":"2024-04-20T02:54:46.470549+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530964+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:40.530964+0000","last_peered":"2024-04-20T02:54:40.530964+0000","last_clean":"2024-04-20T02:54:40.530964+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:40.530964+0000","last_undegraded":"2024-04-20T02:54:40.530964+0000","last_fullsized":"2024-04-20T02:54:40.530964+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178162+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:40.178162+0000","last_peered":"2024-04-20T02:54:40.178162+0000","last_clean":"2024-04-20T02:54:40.178162+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:40.178162+0000","last_undegraded":"2024-04-20T02:54:40.178162+0000","last_fullsized":"2024-04-20T02:54:40.178162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37776,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334064,"statfs":{"total":96636764160,"available":96598081536,"internally_reserved":0,"allocated":2621440,"data_stored":1684472,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.51100000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.372},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.33800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.43099999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70400,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301440,"statfs":{"total":96636764160,"available":96564674560,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53600000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.59799999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.48199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.47099999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.44700000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.50600000000000001}]}]},{"osd":2,"up_from":138,"seq":592705486877,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43380,"kb_used_data":15400,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328460,"statfs":{"total":96636764160,"available":96592343040,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.247}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.29099999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.499}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.56799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.313}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.34300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.53700000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.504}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.498}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.52200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69692,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302148,"statfs":{"total":96636764160,"available":96565399552,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.23699999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.317},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.495}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.504},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574905,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83472,"kb_used_data":48264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288368,"statfs":{"total":96636764160,"available":96551288832,"internally_reserved":0,"allocated":49422336,"data_stored":51697766,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59499999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.57299999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.38300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.36299999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.34599999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509497,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77672,"kb_used_data":38360,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294168,"statfs":{"total":96636764160,"available":96557228032,"internally_reserved":0,"allocated":39280640,"data_stored":40652612,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.30199999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.39700000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.53600000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.502},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.625}]}]},{"osd":4,"up_from":167,"seq":717259538455,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48740,"kb_used_data":20828,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323100,"statfs":{"total":96636764160,"available":96586854400,"internally_reserved":0,"allocated":21327872,"data_stored":21136340,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.72099999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.70899999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.623},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.77200000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.748},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.68200000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53600000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58699999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.69199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:51.915 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:52.244 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:52.246 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:52.258 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-04-20T02:54:51.498218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485844,"kb_used_data":209772,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488876,"statfs":{"total":773094113280,"available":772596609024,"internally_reserved":0,"allocated":214806528,"data_stored":217833050,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.338183"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530769+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:40.530769+0000","last_peered":"2024-04-20T02:54:40.530769+0000","last_clean":"2024-04-20T02:54:40.530769+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:40.530769+0000","last_undegraded":"2024-04-20T02:54:40.530769+0000","last_fullsized":"2024-04-20T02:54:40.530769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":274,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.862079+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:41.862079+0000","last_peered":"2024-04-20T02:54:41.862079+0000","last_clean":"2024-04-20T02:54:41.862079+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:41.862079+0000","last_undegraded":"2024-04-20T02:54:41.862079+0000","last_fullsized":"2024-04-20T02:54:41.862079+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531047+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:40.531047+0000","last_peered":"2024-04-20T02:54:40.531047+0000","last_clean":"2024-04-20T02:54:40.531047+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:40.531047+0000","last_undegraded":"2024-04-20T02:54:40.531047+0000","last_fullsized":"2024-04-20T02:54:40.531047+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":666,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:42.178920+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:42.178920+0000","last_peered":"2024-04-20T02:54:42.178920+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:42.178920+0000","last_undegraded":"2024-04-20T02:54:42.178920+0000","last_fullsized":"2024-04-20T02:54:42.178920+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530766+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:40.530766+0000","last_peered":"2024-04-20T02:54:40.530766+0000","last_clean":"2024-04-20T02:54:40.530766+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:40.530766+0000","last_undegraded":"2024-04-20T02:54:40.530766+0000","last_fullsized":"2024-04-20T02:54:40.530766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":657,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:44.903595+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:44.903595+0000","last_peered":"2024-04-20T02:54:44.903595+0000","last_clean":"2024-04-20T02:54:44.903595+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:44.903595+0000","last_undegraded":"2024-04-20T02:54:44.903595+0000","last_fullsized":"2024-04-20T02:54:44.903595+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":409,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530924+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:40.530924+0000","last_peered":"2024-04-20T02:54:40.530924+0000","last_clean":"2024-04-20T02:54:40.530924+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:40.530924+0000","last_undegraded":"2024-04-20T02:54:40.530924+0000","last_fullsized":"2024-04-20T02:54:40.530924+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531139+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:40.531139+0000","last_peered":"2024-04-20T02:54:40.531139+0000","last_clean":"2024-04-20T02:54:40.531139+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:40.531139+0000","last_undegraded":"2024-04-20T02:54:40.531139+0000","last_fullsized":"2024-04-20T02:54:40.531139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":669,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178166+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:40.178166+0000","last_peered":"2024-04-20T02:54:40.178166+0000","last_clean":"2024-04-20T02:54:40.178166+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:40.178166+0000","last_undegraded":"2024-04-20T02:54:40.178166+0000","last_fullsized":"2024-04-20T02:54:40.178166+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530242+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:40.530242+0000","last_peered":"2024-04-20T02:54:40.530242+0000","last_clean":"2024-04-20T02:54:40.530242+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:40.530242+0000","last_undegraded":"2024-04-20T02:54:40.530242+0000","last_fullsized":"2024-04-20T02:54:40.530242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":681,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:46.470549+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:54:46.470549+0000","last_peered":"2024-04-20T02:54:46.470549+0000","last_clean":"2024-04-20T02:54:46.470549+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:46.470549+0000","last_undegraded":"2024-04-20T02:54:46.470549+0000","last_fullsized":"2024-04-20T02:54:46.470549+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530964+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:40.530964+0000","last_peered":"2024-04-20T02:54:40.530964+0000","last_clean":"2024-04-20T02:54:40.530964+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:40.530964+0000","last_undegraded":"2024-04-20T02:54:40.530964+0000","last_fullsized":"2024-04-20T02:54:40.530964+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178162+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:40.178162+0000","last_peered":"2024-04-20T02:54:40.178162+0000","last_clean":"2024-04-20T02:54:40.178162+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:40.178162+0000","last_undegraded":"2024-04-20T02:54:40.178162+0000","last_fullsized":"2024-04-20T02:54:40.178162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37776,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334064,"statfs":{"total":96636764160,"available":96598081536,"internally_reserved":0,"allocated":2621440,"data_stored":1684472,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.51100000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.372},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.33800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.43099999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70400,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301440,"statfs":{"total":96636764160,"available":96564674560,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53600000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.59799999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.48199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.47099999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.44700000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.50600000000000001}]}]},{"osd":2,"up_from":138,"seq":592705486877,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43380,"kb_used_data":15400,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328460,"statfs":{"total":96636764160,"available":96592343040,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.247}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.29099999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.499}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.56799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.313}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.34300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.53700000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.504}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.498}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.52200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69692,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302148,"statfs":{"total":96636764160,"available":96565399552,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.23699999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.317},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.495}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.504},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574905,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83472,"kb_used_data":48264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288368,"statfs":{"total":96636764160,"available":96551288832,"internally_reserved":0,"allocated":49422336,"data_stored":51697766,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59499999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.57299999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.38300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.36299999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.34599999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509497,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77672,"kb_used_data":38360,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294168,"statfs":{"total":96636764160,"available":96557228032,"internally_reserved":0,"allocated":39280640,"data_stored":40652612,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.30199999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.39700000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.53600000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.502},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.625}]}]},{"osd":4,"up_from":167,"seq":717259538455,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48740,"kb_used_data":20828,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323100,"statfs":{"total":96636764160,"available":96586854400,"internally_reserved":0,"allocated":21327872,"data_stored":21136340,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.72099999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.70899999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.623},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.77200000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.748},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.68200000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53600000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58699999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.69199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:52.260 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:54:52.260 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 666, 'reported_epoch': 230, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:42.178920+0000', 'last_change': '2024-04-20T02:54:42.178857+0000', 'last_active': '2024-04-20T02:54:42.178920+0000', 'last_peered': '2024-04-20T02:54:42.178920+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:42.178920+0000', 'last_undegraded': '2024-04-20T02:54:42.178920+0000', 'last_fullsized': '2024-04-20T02:54:42.178920+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:54:52.260 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:54:52.661 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:52.661 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":285,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":230,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497504256,"bytes_avail":772596609024,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:54:52.675 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:54:52.675 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:52.998 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:53.012 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:53.012 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-04-20T02:54:51.498218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485844,"kb_used_data":209772,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488876,"statfs":{"total":773094113280,"available":772596609024,"internally_reserved":0,"allocated":214806528,"data_stored":217833050,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.338183"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530769+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:40.530769+0000","last_peered":"2024-04-20T02:54:40.530769+0000","last_clean":"2024-04-20T02:54:40.530769+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:40.530769+0000","last_undegraded":"2024-04-20T02:54:40.530769+0000","last_fullsized":"2024-04-20T02:54:40.530769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":274,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.862079+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:41.862079+0000","last_peered":"2024-04-20T02:54:41.862079+0000","last_clean":"2024-04-20T02:54:41.862079+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:41.862079+0000","last_undegraded":"2024-04-20T02:54:41.862079+0000","last_fullsized":"2024-04-20T02:54:41.862079+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531047+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:40.531047+0000","last_peered":"2024-04-20T02:54:40.531047+0000","last_clean":"2024-04-20T02:54:40.531047+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:40.531047+0000","last_undegraded":"2024-04-20T02:54:40.531047+0000","last_fullsized":"2024-04-20T02:54:40.531047+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":666,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:42.178920+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:42.178920+0000","last_peered":"2024-04-20T02:54:42.178920+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:42.178920+0000","last_undegraded":"2024-04-20T02:54:42.178920+0000","last_fullsized":"2024-04-20T02:54:42.178920+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530766+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:40.530766+0000","last_peered":"2024-04-20T02:54:40.530766+0000","last_clean":"2024-04-20T02:54:40.530766+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:40.530766+0000","last_undegraded":"2024-04-20T02:54:40.530766+0000","last_fullsized":"2024-04-20T02:54:40.530766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":657,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:44.903595+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:44.903595+0000","last_peered":"2024-04-20T02:54:44.903595+0000","last_clean":"2024-04-20T02:54:44.903595+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:44.903595+0000","last_undegraded":"2024-04-20T02:54:44.903595+0000","last_fullsized":"2024-04-20T02:54:44.903595+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":409,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530924+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:40.530924+0000","last_peered":"2024-04-20T02:54:40.530924+0000","last_clean":"2024-04-20T02:54:40.530924+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:40.530924+0000","last_undegraded":"2024-04-20T02:54:40.530924+0000","last_fullsized":"2024-04-20T02:54:40.530924+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531139+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:40.531139+0000","last_peered":"2024-04-20T02:54:40.531139+0000","last_clean":"2024-04-20T02:54:40.531139+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:40.531139+0000","last_undegraded":"2024-04-20T02:54:40.531139+0000","last_fullsized":"2024-04-20T02:54:40.531139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":669,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178166+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:40.178166+0000","last_peered":"2024-04-20T02:54:40.178166+0000","last_clean":"2024-04-20T02:54:40.178166+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:40.178166+0000","last_undegraded":"2024-04-20T02:54:40.178166+0000","last_fullsized":"2024-04-20T02:54:40.178166+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530242+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:40.530242+0000","last_peered":"2024-04-20T02:54:40.530242+0000","last_clean":"2024-04-20T02:54:40.530242+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:40.530242+0000","last_undegraded":"2024-04-20T02:54:40.530242+0000","last_fullsized":"2024-04-20T02:54:40.530242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":681,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:46.470549+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:54:46.470549+0000","last_peered":"2024-04-20T02:54:46.470549+0000","last_clean":"2024-04-20T02:54:46.470549+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:46.470549+0000","last_undegraded":"2024-04-20T02:54:46.470549+0000","last_fullsized":"2024-04-20T02:54:46.470549+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530964+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:40.530964+0000","last_peered":"2024-04-20T02:54:40.530964+0000","last_clean":"2024-04-20T02:54:40.530964+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:40.530964+0000","last_undegraded":"2024-04-20T02:54:40.530964+0000","last_fullsized":"2024-04-20T02:54:40.530964+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178162+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:40.178162+0000","last_peered":"2024-04-20T02:54:40.178162+0000","last_clean":"2024-04-20T02:54:40.178162+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:40.178162+0000","last_undegraded":"2024-04-20T02:54:40.178162+0000","last_fullsized":"2024-04-20T02:54:40.178162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37776,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334064,"statfs":{"total":96636764160,"available":96598081536,"internally_reserved":0,"allocated":2621440,"data_stored":1684472,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.51100000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.372},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.33800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.43099999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70400,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301440,"statfs":{"total":96636764160,"available":96564674560,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53600000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.59799999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.48199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.47099999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.44700000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.50600000000000001}]}]},{"osd":2,"up_from":138,"seq":592705486877,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43380,"kb_used_data":15400,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328460,"statfs":{"total":96636764160,"available":96592343040,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.38800000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.247}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.29099999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.499}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.56799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.313}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.34300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.53700000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.504}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.498}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.52200000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69692,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302148,"statfs":{"total":96636764160,"available":96565399552,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.23699999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.317},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.495}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.504},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574905,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83472,"kb_used_data":48264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288368,"statfs":{"total":96636764160,"available":96551288832,"internally_reserved":0,"allocated":49422336,"data_stored":51697766,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59499999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.57299999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.38300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.36299999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.34599999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509497,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77672,"kb_used_data":38360,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294168,"statfs":{"total":96636764160,"available":96557228032,"internally_reserved":0,"allocated":39280640,"data_stored":40652612,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.30199999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.39700000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.53600000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.502},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.625}]}]},{"osd":4,"up_from":167,"seq":717259538455,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48740,"kb_used_data":20828,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323100,"statfs":{"total":96636764160,"available":96586854400,"internally_reserved":0,"allocated":21327872,"data_stored":21136340,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.72099999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.70899999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.623},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.77200000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.748},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.68200000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53600000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58699999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.69199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:56.014 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:56.358 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:56.360 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:56.372 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-04-20T02:54:56.235881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485844,"kb_used_data":209772,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488876,"statfs":{"total":773094113280,"available":772596609024,"internally_reserved":0,"allocated":214806528,"data_stored":217833050,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.738774"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530769+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:40.530769+0000","last_peered":"2024-04-20T02:54:40.530769+0000","last_clean":"2024-04-20T02:54:40.530769+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:40.530769+0000","last_undegraded":"2024-04-20T02:54:40.530769+0000","last_fullsized":"2024-04-20T02:54:40.530769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":274,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.862079+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:41.862079+0000","last_peered":"2024-04-20T02:54:41.862079+0000","last_clean":"2024-04-20T02:54:41.862079+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:41.862079+0000","last_undegraded":"2024-04-20T02:54:41.862079+0000","last_fullsized":"2024-04-20T02:54:41.862079+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531047+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:40.531047+0000","last_peered":"2024-04-20T02:54:40.531047+0000","last_clean":"2024-04-20T02:54:40.531047+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:40.531047+0000","last_undegraded":"2024-04-20T02:54:40.531047+0000","last_fullsized":"2024-04-20T02:54:40.531047+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":666,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:42.178920+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:42.178920+0000","last_peered":"2024-04-20T02:54:42.178920+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:42.178920+0000","last_undegraded":"2024-04-20T02:54:42.178920+0000","last_fullsized":"2024-04-20T02:54:42.178920+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530766+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:40.530766+0000","last_peered":"2024-04-20T02:54:40.530766+0000","last_clean":"2024-04-20T02:54:40.530766+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:40.530766+0000","last_undegraded":"2024-04-20T02:54:40.530766+0000","last_fullsized":"2024-04-20T02:54:40.530766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":657,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:44.903595+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:44.903595+0000","last_peered":"2024-04-20T02:54:44.903595+0000","last_clean":"2024-04-20T02:54:44.903595+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:44.903595+0000","last_undegraded":"2024-04-20T02:54:44.903595+0000","last_fullsized":"2024-04-20T02:54:44.903595+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":409,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530924+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:40.530924+0000","last_peered":"2024-04-20T02:54:40.530924+0000","last_clean":"2024-04-20T02:54:40.530924+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:40.530924+0000","last_undegraded":"2024-04-20T02:54:40.530924+0000","last_fullsized":"2024-04-20T02:54:40.530924+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531139+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:40.531139+0000","last_peered":"2024-04-20T02:54:40.531139+0000","last_clean":"2024-04-20T02:54:40.531139+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:40.531139+0000","last_undegraded":"2024-04-20T02:54:40.531139+0000","last_fullsized":"2024-04-20T02:54:40.531139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":669,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178166+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:40.178166+0000","last_peered":"2024-04-20T02:54:40.178166+0000","last_clean":"2024-04-20T02:54:40.178166+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:40.178166+0000","last_undegraded":"2024-04-20T02:54:40.178166+0000","last_fullsized":"2024-04-20T02:54:40.178166+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530242+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:40.530242+0000","last_peered":"2024-04-20T02:54:40.530242+0000","last_clean":"2024-04-20T02:54:40.530242+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:40.530242+0000","last_undegraded":"2024-04-20T02:54:40.530242+0000","last_fullsized":"2024-04-20T02:54:40.530242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":681,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:46.470549+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:54:46.470549+0000","last_peered":"2024-04-20T02:54:46.470549+0000","last_clean":"2024-04-20T02:54:46.470549+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:46.470549+0000","last_undegraded":"2024-04-20T02:54:46.470549+0000","last_fullsized":"2024-04-20T02:54:46.470549+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530964+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:40.530964+0000","last_peered":"2024-04-20T02:54:40.530964+0000","last_clean":"2024-04-20T02:54:40.530964+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:40.530964+0000","last_undegraded":"2024-04-20T02:54:40.530964+0000","last_fullsized":"2024-04-20T02:54:40.530964+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178162+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:40.178162+0000","last_peered":"2024-04-20T02:54:40.178162+0000","last_clean":"2024-04-20T02:54:40.178162+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:40.178162+0000","last_undegraded":"2024-04-20T02:54:40.178162+0000","last_fullsized":"2024-04-20T02:54:40.178162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37776,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334064,"statfs":{"total":96636764160,"available":96598081536,"internally_reserved":0,"allocated":2621440,"data_stored":1684472,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.624},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.68400000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.5}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.72899999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.76400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70400,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301440,"statfs":{"total":96636764160,"available":96564674560,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59499999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.503}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51400000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.53600000000000003}]}]},{"osd":2,"up_from":138,"seq":592705486878,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43380,"kb_used_data":15400,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328460,"statfs":{"total":96636764160,"available":96592343040,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.374},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.48699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.51300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.65000000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.57699999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.65900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.61699999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.627}]}]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69692,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302148,"statfs":{"total":96636764160,"available":96565399552,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.251},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.36899999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.38}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67300000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.71699999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83472,"kb_used_data":48264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288368,"statfs":{"total":96636764160,"available":96551288832,"internally_reserved":0,"allocated":49422336,"data_stored":51697766,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.41999999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.376}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52200000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.432},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.39400000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.35999999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.45600000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509498,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77672,"kb_used_data":38360,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294168,"statfs":{"total":96636764160,"available":96557228032,"internally_reserved":0,"allocated":39280640,"data_stored":40652612,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.35699999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.374}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.34100000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.502},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.64000000000000001}]}]},{"osd":4,"up_from":167,"seq":717259538456,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48740,"kb_used_data":20828,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323100,"statfs":{"total":96636764160,"available":96586854400,"internally_reserved":0,"allocated":21327872,"data_stored":21136340,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.629},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.47699999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.439},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.495}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.42199999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:56.374 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:56.710 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:56.712 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:56.724 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-04-20T02:54:56.235881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485844,"kb_used_data":209772,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488876,"statfs":{"total":773094113280,"available":772596609024,"internally_reserved":0,"allocated":214806528,"data_stored":217833050,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.738774"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530769+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:40.530769+0000","last_peered":"2024-04-20T02:54:40.530769+0000","last_clean":"2024-04-20T02:54:40.530769+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:40.530769+0000","last_undegraded":"2024-04-20T02:54:40.530769+0000","last_fullsized":"2024-04-20T02:54:40.530769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":274,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.862079+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:41.862079+0000","last_peered":"2024-04-20T02:54:41.862079+0000","last_clean":"2024-04-20T02:54:41.862079+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:41.862079+0000","last_undegraded":"2024-04-20T02:54:41.862079+0000","last_fullsized":"2024-04-20T02:54:41.862079+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531047+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:40.531047+0000","last_peered":"2024-04-20T02:54:40.531047+0000","last_clean":"2024-04-20T02:54:40.531047+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:40.531047+0000","last_undegraded":"2024-04-20T02:54:40.531047+0000","last_fullsized":"2024-04-20T02:54:40.531047+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":666,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:42.178920+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:42.178920+0000","last_peered":"2024-04-20T02:54:42.178920+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:42.178920+0000","last_undegraded":"2024-04-20T02:54:42.178920+0000","last_fullsized":"2024-04-20T02:54:42.178920+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530766+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:40.530766+0000","last_peered":"2024-04-20T02:54:40.530766+0000","last_clean":"2024-04-20T02:54:40.530766+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:40.530766+0000","last_undegraded":"2024-04-20T02:54:40.530766+0000","last_fullsized":"2024-04-20T02:54:40.530766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":657,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:44.903595+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:44.903595+0000","last_peered":"2024-04-20T02:54:44.903595+0000","last_clean":"2024-04-20T02:54:44.903595+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:44.903595+0000","last_undegraded":"2024-04-20T02:54:44.903595+0000","last_fullsized":"2024-04-20T02:54:44.903595+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":409,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530924+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:40.530924+0000","last_peered":"2024-04-20T02:54:40.530924+0000","last_clean":"2024-04-20T02:54:40.530924+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:40.530924+0000","last_undegraded":"2024-04-20T02:54:40.530924+0000","last_fullsized":"2024-04-20T02:54:40.530924+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531139+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:40.531139+0000","last_peered":"2024-04-20T02:54:40.531139+0000","last_clean":"2024-04-20T02:54:40.531139+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:40.531139+0000","last_undegraded":"2024-04-20T02:54:40.531139+0000","last_fullsized":"2024-04-20T02:54:40.531139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":669,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178166+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:40.178166+0000","last_peered":"2024-04-20T02:54:40.178166+0000","last_clean":"2024-04-20T02:54:40.178166+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:40.178166+0000","last_undegraded":"2024-04-20T02:54:40.178166+0000","last_fullsized":"2024-04-20T02:54:40.178166+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530242+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:40.530242+0000","last_peered":"2024-04-20T02:54:40.530242+0000","last_clean":"2024-04-20T02:54:40.530242+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:40.530242+0000","last_undegraded":"2024-04-20T02:54:40.530242+0000","last_fullsized":"2024-04-20T02:54:40.530242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":681,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:46.470549+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:54:46.470549+0000","last_peered":"2024-04-20T02:54:46.470549+0000","last_clean":"2024-04-20T02:54:46.470549+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:46.470549+0000","last_undegraded":"2024-04-20T02:54:46.470549+0000","last_fullsized":"2024-04-20T02:54:46.470549+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530964+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:40.530964+0000","last_peered":"2024-04-20T02:54:40.530964+0000","last_clean":"2024-04-20T02:54:40.530964+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:40.530964+0000","last_undegraded":"2024-04-20T02:54:40.530964+0000","last_fullsized":"2024-04-20T02:54:40.530964+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178162+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:40.178162+0000","last_peered":"2024-04-20T02:54:40.178162+0000","last_clean":"2024-04-20T02:54:40.178162+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:40.178162+0000","last_undegraded":"2024-04-20T02:54:40.178162+0000","last_fullsized":"2024-04-20T02:54:40.178162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37776,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334064,"statfs":{"total":96636764160,"available":96598081536,"internally_reserved":0,"allocated":2621440,"data_stored":1684472,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.624},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.68400000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.5}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.72899999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.76400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70400,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301440,"statfs":{"total":96636764160,"available":96564674560,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59499999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.503}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51400000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.53600000000000003}]}]},{"osd":2,"up_from":138,"seq":592705486878,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43380,"kb_used_data":15400,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328460,"statfs":{"total":96636764160,"available":96592343040,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.374},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.48699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.51300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.65000000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.57699999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.65900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.61699999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.627}]}]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69692,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302148,"statfs":{"total":96636764160,"available":96565399552,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.251},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.36899999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.38}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67300000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.71699999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83472,"kb_used_data":48264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288368,"statfs":{"total":96636764160,"available":96551288832,"internally_reserved":0,"allocated":49422336,"data_stored":51697766,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.41999999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.376}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52200000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.432},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.39400000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.35999999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.45600000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509498,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77672,"kb_used_data":38360,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294168,"statfs":{"total":96636764160,"available":96557228032,"internally_reserved":0,"allocated":39280640,"data_stored":40652612,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.35699999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.374}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.34100000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.502},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.64000000000000001}]}]},{"osd":4,"up_from":167,"seq":717259538456,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48740,"kb_used_data":20828,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323100,"statfs":{"total":96636764160,"available":96586854400,"internally_reserved":0,"allocated":21327872,"data_stored":21136340,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.629},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.47699999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.439},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.495}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.42199999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:54:56.726 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:54:56.726 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 666, 'reported_epoch': 230, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:42.178920+0000', 'last_change': '2024-04-20T02:54:42.178857+0000', 'last_active': '2024-04-20T02:54:42.178920+0000', 'last_peered': '2024-04-20T02:54:42.178920+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:42.178920+0000', 'last_undegraded': '2024-04-20T02:54:42.178920+0000', 'last_fullsized': '2024-04-20T02:54:42.178920+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:54:56.726 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:54:57.128 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:57.128 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":289,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":231,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497504256,"bytes_avail":772596609024,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:54:57.142 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:54:57.142 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:54:57.472 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:54:57.473 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:54:57.485 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-04-20T02:54:57.237106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485844,"kb_used_data":209772,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488876,"statfs":{"total":773094113280,"available":772596609024,"internally_reserved":0,"allocated":214806528,"data_stored":217833050,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.739611"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530769+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:40.530769+0000","last_peered":"2024-04-20T02:54:40.530769+0000","last_clean":"2024-04-20T02:54:40.530769+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:40.530769+0000","last_undegraded":"2024-04-20T02:54:40.530769+0000","last_fullsized":"2024-04-20T02:54:40.530769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":274,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.862079+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:41.862079+0000","last_peered":"2024-04-20T02:54:41.862079+0000","last_clean":"2024-04-20T02:54:41.862079+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:41.862079+0000","last_undegraded":"2024-04-20T02:54:41.862079+0000","last_fullsized":"2024-04-20T02:54:41.862079+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531047+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:40.531047+0000","last_peered":"2024-04-20T02:54:40.531047+0000","last_clean":"2024-04-20T02:54:40.531047+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:40.531047+0000","last_undegraded":"2024-04-20T02:54:40.531047+0000","last_fullsized":"2024-04-20T02:54:40.531047+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":666,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:42.178920+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:42.178920+0000","last_peered":"2024-04-20T02:54:42.178920+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:42.178920+0000","last_undegraded":"2024-04-20T02:54:42.178920+0000","last_fullsized":"2024-04-20T02:54:42.178920+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530766+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:40.530766+0000","last_peered":"2024-04-20T02:54:40.530766+0000","last_clean":"2024-04-20T02:54:40.530766+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:40.530766+0000","last_undegraded":"2024-04-20T02:54:40.530766+0000","last_fullsized":"2024-04-20T02:54:40.530766+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":657,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:44.903595+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:44.903595+0000","last_peered":"2024-04-20T02:54:44.903595+0000","last_clean":"2024-04-20T02:54:44.903595+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:44.903595+0000","last_undegraded":"2024-04-20T02:54:44.903595+0000","last_fullsized":"2024-04-20T02:54:44.903595+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":409,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530924+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:40.530924+0000","last_peered":"2024-04-20T02:54:40.530924+0000","last_clean":"2024-04-20T02:54:40.530924+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:40.530924+0000","last_undegraded":"2024-04-20T02:54:40.530924+0000","last_fullsized":"2024-04-20T02:54:40.530924+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531139+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:40.531139+0000","last_peered":"2024-04-20T02:54:40.531139+0000","last_clean":"2024-04-20T02:54:40.531139+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:40.531139+0000","last_undegraded":"2024-04-20T02:54:40.531139+0000","last_fullsized":"2024-04-20T02:54:40.531139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":669,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178166+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:40.178166+0000","last_peered":"2024-04-20T02:54:40.178166+0000","last_clean":"2024-04-20T02:54:40.178166+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:40.178166+0000","last_undegraded":"2024-04-20T02:54:40.178166+0000","last_fullsized":"2024-04-20T02:54:40.178166+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530242+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:40.530242+0000","last_peered":"2024-04-20T02:54:40.530242+0000","last_clean":"2024-04-20T02:54:40.530242+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:40.530242+0000","last_undegraded":"2024-04-20T02:54:40.530242+0000","last_fullsized":"2024-04-20T02:54:40.530242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":681,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:46.470549+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:54:46.470549+0000","last_peered":"2024-04-20T02:54:46.470549+0000","last_clean":"2024-04-20T02:54:46.470549+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:46.470549+0000","last_undegraded":"2024-04-20T02:54:46.470549+0000","last_fullsized":"2024-04-20T02:54:46.470549+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":217,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530964+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:40.530964+0000","last_peered":"2024-04-20T02:54:40.530964+0000","last_clean":"2024-04-20T02:54:40.530964+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:40.530964+0000","last_undegraded":"2024-04-20T02:54:40.530964+0000","last_fullsized":"2024-04-20T02:54:40.530964+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.178162+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:40.178162+0000","last_peered":"2024-04-20T02:54:40.178162+0000","last_clean":"2024-04-20T02:54:40.178162+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:40.178162+0000","last_undegraded":"2024-04-20T02:54:40.178162+0000","last_fullsized":"2024-04-20T02:54:40.178162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37776,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334064,"statfs":{"total":96636764160,"available":96598081536,"internally_reserved":0,"allocated":2621440,"data_stored":1684472,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.624},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.68400000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.5}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.72899999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.76400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70400,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301440,"statfs":{"total":96636764160,"available":96564674560,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59499999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.503}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51400000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.53600000000000003}]}]},{"osd":2,"up_from":138,"seq":592705486878,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43380,"kb_used_data":15400,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328460,"statfs":{"total":96636764160,"available":96592343040,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.374},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.48699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.51300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.65000000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.57699999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.65900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.61699999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53800000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.627}]}]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69692,"kb_used_data":34480,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302148,"statfs":{"total":96636764160,"available":96565399552,"internally_reserved":0,"allocated":35307520,"data_stored":36206194,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.251},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.36899999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.38}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.29799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67300000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68700000000000006}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.71699999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574906,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83472,"kb_used_data":48264,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288368,"statfs":{"total":96636764160,"available":96551288832,"internally_reserved":0,"allocated":49422336,"data_stored":51697766,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.41999999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.376}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52200000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.432},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.39400000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.35999999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.45600000000000002}]}]},{"osd":0,"up_from":15,"seq":64424509498,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77672,"kb_used_data":38360,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294168,"statfs":{"total":96636764160,"available":96557228032,"internally_reserved":0,"allocated":39280640,"data_stored":40652612,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.35699999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.374}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.34100000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.502},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.64000000000000001}]}]},{"osd":4,"up_from":167,"seq":717259538456,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48740,"kb_used_data":20828,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323100,"statfs":{"total":96636764160,"available":96586854400,"internally_reserved":0,"allocated":21327872,"data_stored":21136340,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.629},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.47699999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.439},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.495}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.42199999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:00.488 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:00.827 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:00.829 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:00.840 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-04-20T02:54:59.500037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485900,"kb_used_data":209832,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488820,"statfs":{"total":773094113280,"available":772596551680,"internally_reserved":0,"allocated":214867968,"data_stored":217857120,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001405"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":206,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-20T02:54:56.239431+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:56.239431+0000","last_peered":"2024-04-20T02:54:56.239431+0000","last_clean":"2024-04-20T02:54:56.239431+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:56.239431+0000","last_undegraded":"2024-04-20T02:54:56.239431+0000","last_fullsized":"2024-04-20T02:54:56.239431+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":275,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:47.632212+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:47.632212+0000","last_peered":"2024-04-20T02:54:47.632212+0000","last_clean":"2024-04-20T02:54:47.632212+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:47.632212+0000","last_undegraded":"2024-04-20T02:54:47.632212+0000","last_fullsized":"2024-04-20T02:54:47.632212+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.531047+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:40.531047+0000","last_peered":"2024-04-20T02:54:40.531047+0000","last_clean":"2024-04-20T02:54:40.531047+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:40.531047+0000","last_undegraded":"2024-04-20T02:54:40.531047+0000","last_fullsized":"2024-04-20T02:54:40.531047+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":666,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:42.178920+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:42.178920+0000","last_peered":"2024-04-20T02:54:42.178920+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:42.178920+0000","last_undegraded":"2024-04-20T02:54:42.178920+0000","last_fullsized":"2024-04-20T02:54:42.178920+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-20T02:54:56.239348+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:56.239348+0000","last_peered":"2024-04-20T02:54:56.239348+0000","last_clean":"2024-04-20T02:54:56.239348+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:56.239348+0000","last_undegraded":"2024-04-20T02:54:56.239348+0000","last_fullsized":"2024-04-20T02:54:56.239348+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":659,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-20T02:54:56.241681+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:56.241681+0000","last_peered":"2024-04-20T02:54:56.241681+0000","last_clean":"2024-04-20T02:54:56.241681+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:56.241681+0000","last_undegraded":"2024-04-20T02:54:56.241681+0000","last_fullsized":"2024-04-20T02:54:56.241681+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":409,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530924+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:40.530924+0000","last_peered":"2024-04-20T02:54:40.530924+0000","last_clean":"2024-04-20T02:54:40.530924+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:40.530924+0000","last_undegraded":"2024-04-20T02:54:40.530924+0000","last_fullsized":"2024-04-20T02:54:40.530924+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":251,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:42.084594+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:42.084594+0000","last_peered":"2024-04-20T02:54:42.084594+0000","last_clean":"2024-04-20T02:54:42.084594+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:42.084594+0000","last_undegraded":"2024-04-20T02:54:42.084594+0000","last_fullsized":"2024-04-20T02:54:42.084594+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":670,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.161691+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:41.161691+0000","last_peered":"2024-04-20T02:54:41.161691+0000","last_clean":"2024-04-20T02:54:41.161691+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:41.161691+0000","last_undegraded":"2024-04-20T02:54:41.161691+0000","last_fullsized":"2024-04-20T02:54:41.161691+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-04-20T02:54:40.530242+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:40.530242+0000","last_peered":"2024-04-20T02:54:40.530242+0000","last_clean":"2024-04-20T02:54:40.530242+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:40.530242+0000","last_undegraded":"2024-04-20T02:54:40.530242+0000","last_fullsized":"2024-04-20T02:54:40.530242+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":681,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:46.470549+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:54:46.470549+0000","last_peered":"2024-04-20T02:54:46.470549+0000","last_clean":"2024-04-20T02:54:46.470549+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:46.470549+0000","last_undegraded":"2024-04-20T02:54:46.470549+0000","last_fullsized":"2024-04-20T02:54:46.470549+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":218,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:42.084590+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:42.084590+0000","last_peered":"2024-04-20T02:54:42.084590+0000","last_clean":"2024-04-20T02:54:42.084590+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:42.084590+0000","last_undegraded":"2024-04-20T02:54:42.084590+0000","last_fullsized":"2024-04-20T02:54:42.084590+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":246,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.161689+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:41.161689+0000","last_peered":"2024-04-20T02:54:41.161689+0000","last_clean":"2024-04-20T02:54:41.161689+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:41.161689+0000","last_undegraded":"2024-04-20T02:54:41.161689+0000","last_fullsized":"2024-04-20T02:54:41.161689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37776,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334064,"statfs":{"total":96636764160,"available":96598081536,"internally_reserved":0,"allocated":2621440,"data_stored":1684472,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.624},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.68400000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.5}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.72899999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.76400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70420,"kb_used_data":34504,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301420,"statfs":{"total":96636764160,"available":96564654080,"internally_reserved":0,"allocated":35332096,"data_stored":36215822,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.626}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.50800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.443},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54600000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.58799999999999997}]}]},{"osd":2,"up_from":138,"seq":592705486878,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43380,"kb_used_data":15400,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328460,"statfs":{"total":96636764160,"available":96592343040,"internally_reserved":0,"allocated":15769600,"data_stored":15124736,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.374},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.48699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.51300000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.65000000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.57699999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.65900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54720,"kb_used_data":15412,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317120,"statfs":{"total":96636764160,"available":96580730880,"internally_reserved":0,"allocated":15781888,"data_stored":15129550,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.316}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.376}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.438}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.36599999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.502},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.42799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69704,"kb_used_data":34492,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302136,"statfs":{"total":96636764160,"available":96565387264,"internally_reserved":0,"allocated":35319808,"data_stored":36211008,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.627}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83488,"kb_used_data":48276,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288352,"statfs":{"total":96636764160,"available":96551272448,"internally_reserved":0,"allocated":49434624,"data_stored":51702580,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.55600000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.5},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.75800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.83299999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.66100000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.746}]}]},{"osd":0,"up_from":15,"seq":64424509498,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77672,"kb_used_data":38360,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294168,"statfs":{"total":96636764160,"available":96557228032,"internally_reserved":0,"allocated":39280640,"data_stored":40652612,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.35699999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.374}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.34100000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:53:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.379,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.502},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.64000000000000001}]}]},{"osd":4,"up_from":167,"seq":717259538456,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48740,"kb_used_data":20828,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323100,"statfs":{"total":96636764160,"available":96586854400,"internally_reserved":0,"allocated":21327872,"data_stored":21136340,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.58399999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.629},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.47699999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.439},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.495}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.42199999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:00.843 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:01.171 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:01.183 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-04-20T02:55:01.161885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486064,"kb_used_data":209976,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488656,"statfs":{"total":773094113280,"available":772596383744,"internally_reserved":0,"allocated":215015424,"data_stored":217914888,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.662805"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":206,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-20T02:54:56.239431+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:56.239431+0000","last_peered":"2024-04-20T02:54:56.239431+0000","last_clean":"2024-04-20T02:54:56.239431+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:56.239431+0000","last_undegraded":"2024-04-20T02:54:56.239431+0000","last_fullsized":"2024-04-20T02:54:56.239431+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":275,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:47.632212+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:47.632212+0000","last_peered":"2024-04-20T02:54:47.632212+0000","last_clean":"2024-04-20T02:54:47.632212+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:47.632212+0000","last_undegraded":"2024-04-20T02:54:47.632212+0000","last_fullsized":"2024-04-20T02:54:47.632212+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":276,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.386675+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:58.386675+0000","last_peered":"2024-04-20T02:54:58.386675+0000","last_clean":"2024-04-20T02:54:58.386675+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:58.386675+0000","last_undegraded":"2024-04-20T02:54:58.386675+0000","last_fullsized":"2024-04-20T02:54:58.386675+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":669,"reported_epoch":232,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:57.286187+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:57.286187+0000","last_peered":"2024-04-20T02:54:57.286187+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:57.286187+0000","last_undegraded":"2024-04-20T02:54:57.286187+0000","last_fullsized":"2024-04-20T02:54:57.286187+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-20T02:54:56.239348+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:56.239348+0000","last_peered":"2024-04-20T02:54:56.239348+0000","last_clean":"2024-04-20T02:54:56.239348+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:56.239348+0000","last_undegraded":"2024-04-20T02:54:56.239348+0000","last_fullsized":"2024-04-20T02:54:56.239348+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":659,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-20T02:54:56.241681+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:56.241681+0000","last_peered":"2024-04-20T02:54:56.241681+0000","last_clean":"2024-04-20T02:54:56.241681+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:56.241681+0000","last_undegraded":"2024-04-20T02:54:56.241681+0000","last_fullsized":"2024-04-20T02:54:56.241681+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":412,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.386696+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:58.386696+0000","last_peered":"2024-04-20T02:54:58.386696+0000","last_clean":"2024-04-20T02:54:58.386696+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:58.386696+0000","last_undegraded":"2024-04-20T02:54:58.386696+0000","last_fullsized":"2024-04-20T02:54:58.386696+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":251,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:42.084594+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:42.084594+0000","last_peered":"2024-04-20T02:54:42.084594+0000","last_clean":"2024-04-20T02:54:42.084594+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:42.084594+0000","last_undegraded":"2024-04-20T02:54:42.084594+0000","last_fullsized":"2024-04-20T02:54:42.084594+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":670,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.161691+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:41.161691+0000","last_peered":"2024-04-20T02:54:41.161691+0000","last_clean":"2024-04-20T02:54:41.161691+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:41.161691+0000","last_undegraded":"2024-04-20T02:54:41.161691+0000","last_fullsized":"2024-04-20T02:54:41.161691+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.278468+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:58.278468+0000","last_peered":"2024-04-20T02:54:58.278468+0000","last_clean":"2024-04-20T02:54:58.278468+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:58.278468+0000","last_undegraded":"2024-04-20T02:54:58.278468+0000","last_fullsized":"2024-04-20T02:54:58.278468+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":684,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:57.238401+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:54:57.238401+0000","last_peered":"2024-04-20T02:54:57.238401+0000","last_clean":"2024-04-20T02:54:57.238401+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:57.238401+0000","last_undegraded":"2024-04-20T02:54:57.238401+0000","last_fullsized":"2024-04-20T02:54:57.238401+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":218,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:42.084590+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:42.084590+0000","last_peered":"2024-04-20T02:54:42.084590+0000","last_clean":"2024-04-20T02:54:42.084590+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:42.084590+0000","last_undegraded":"2024-04-20T02:54:42.084590+0000","last_fullsized":"2024-04-20T02:54:42.084590+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":246,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.161689+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:41.161689+0000","last_peered":"2024-04-20T02:54:41.161689+0000","last_clean":"2024-04-20T02:54:41.161689+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:41.161689+0000","last_undegraded":"2024-04-20T02:54:41.161689+0000","last_fullsized":"2024-04-20T02:54:41.161689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37816,"kb_used_data":2596,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334024,"statfs":{"total":96636764160,"available":96598040576,"internally_reserved":0,"allocated":2658304,"data_stored":1698914,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.749},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.77600000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.73099999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.69299999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.78700000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.80100000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70420,"kb_used_data":34504,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301420,"statfs":{"total":96636764160,"available":96564654080,"internally_reserved":0,"allocated":35332096,"data_stored":36215822,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.626}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.50800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.443},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54600000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.58799999999999997}]}]},{"osd":2,"up_from":138,"seq":592705486879,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43424,"kb_used_data":15436,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328416,"statfs":{"total":96636764160,"available":96592297984,"internally_reserved":0,"allocated":15806464,"data_stored":15139178,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.44}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.621}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.60699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.68999999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.67400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54720,"kb_used_data":15412,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317120,"statfs":{"total":96636764160,"available":96580730880,"internally_reserved":0,"allocated":15781888,"data_stored":15129550,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.316}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.376}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.438}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.36599999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.502},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.42799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69704,"kb_used_data":34492,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302136,"statfs":{"total":96636764160,"available":96565387264,"internally_reserved":0,"allocated":35319808,"data_stored":36211008,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.627}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83488,"kb_used_data":48276,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288352,"statfs":{"total":96636764160,"available":96551272448,"internally_reserved":0,"allocated":49434624,"data_stored":51702580,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.55600000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.5},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.75800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.83299999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.66100000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.746}]}]},{"osd":0,"up_from":15,"seq":64424509499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77704,"kb_used_data":38396,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294136,"statfs":{"total":96636764160,"available":96557195264,"internally_reserved":0,"allocated":39317504,"data_stored":40667054,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.443}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.433}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.626}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.67100000000000004}]}]},{"osd":4,"up_from":167,"seq":717259538457,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48788,"kb_used_data":20864,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323052,"statfs":{"total":96636764160,"available":96586805248,"internally_reserved":0,"allocated":21364736,"data_stored":21150782,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.501}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.44500000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.38}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:01.185 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:55:01.185 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 669, 'reported_epoch': 232, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:57.286187+0000', 'last_change': '2024-04-20T02:54:42.178857+0000', 'last_active': '2024-04-20T02:54:57.286187+0000', 'last_peered': '2024-04-20T02:54:57.286187+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:57.286187+0000', 'last_undegraded': '2024-04-20T02:54:57.286187+0000', 'last_fullsized': '2024-04-20T02:54:57.286187+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:55:01.185 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:55:01.592 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:01.592 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":294,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":234,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497561600,"bytes_avail":772596551680,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:55:01.607 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:55:01.607 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:01.930 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:01.932 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:01.943 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-04-20T02:55:01.500458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486064,"kb_used_data":209976,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488656,"statfs":{"total":773094113280,"available":772596383744,"internally_reserved":0,"allocated":215015424,"data_stored":217914888,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.264576"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":206,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-20T02:54:56.239431+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:56.239431+0000","last_peered":"2024-04-20T02:54:56.239431+0000","last_clean":"2024-04-20T02:54:56.239431+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:56.239431+0000","last_undegraded":"2024-04-20T02:54:56.239431+0000","last_fullsized":"2024-04-20T02:54:56.239431+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":275,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:47.632212+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:47.632212+0000","last_peered":"2024-04-20T02:54:47.632212+0000","last_clean":"2024-04-20T02:54:47.632212+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:47.632212+0000","last_undegraded":"2024-04-20T02:54:47.632212+0000","last_fullsized":"2024-04-20T02:54:47.632212+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":276,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.386675+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:58.386675+0000","last_peered":"2024-04-20T02:54:58.386675+0000","last_clean":"2024-04-20T02:54:58.386675+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:58.386675+0000","last_undegraded":"2024-04-20T02:54:58.386675+0000","last_fullsized":"2024-04-20T02:54:58.386675+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":669,"reported_epoch":232,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:57.286187+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:57.286187+0000","last_peered":"2024-04-20T02:54:57.286187+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:57.286187+0000","last_undegraded":"2024-04-20T02:54:57.286187+0000","last_fullsized":"2024-04-20T02:54:57.286187+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-20T02:54:56.239348+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:56.239348+0000","last_peered":"2024-04-20T02:54:56.239348+0000","last_clean":"2024-04-20T02:54:56.239348+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:56.239348+0000","last_undegraded":"2024-04-20T02:54:56.239348+0000","last_fullsized":"2024-04-20T02:54:56.239348+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":659,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-04-20T02:54:56.241681+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:56.241681+0000","last_peered":"2024-04-20T02:54:56.241681+0000","last_clean":"2024-04-20T02:54:56.241681+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:56.241681+0000","last_undegraded":"2024-04-20T02:54:56.241681+0000","last_fullsized":"2024-04-20T02:54:56.241681+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":412,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.386696+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:58.386696+0000","last_peered":"2024-04-20T02:54:58.386696+0000","last_clean":"2024-04-20T02:54:58.386696+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:58.386696+0000","last_undegraded":"2024-04-20T02:54:58.386696+0000","last_fullsized":"2024-04-20T02:54:58.386696+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":251,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:42.084594+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:42.084594+0000","last_peered":"2024-04-20T02:54:42.084594+0000","last_clean":"2024-04-20T02:54:42.084594+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:42.084594+0000","last_undegraded":"2024-04-20T02:54:42.084594+0000","last_fullsized":"2024-04-20T02:54:42.084594+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":670,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.161691+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:41.161691+0000","last_peered":"2024-04-20T02:54:41.161691+0000","last_clean":"2024-04-20T02:54:41.161691+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:41.161691+0000","last_undegraded":"2024-04-20T02:54:41.161691+0000","last_fullsized":"2024-04-20T02:54:41.161691+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.278468+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:58.278468+0000","last_peered":"2024-04-20T02:54:58.278468+0000","last_clean":"2024-04-20T02:54:58.278468+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:58.278468+0000","last_undegraded":"2024-04-20T02:54:58.278468+0000","last_fullsized":"2024-04-20T02:54:58.278468+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":684,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:57.238401+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:54:57.238401+0000","last_peered":"2024-04-20T02:54:57.238401+0000","last_clean":"2024-04-20T02:54:57.238401+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:57.238401+0000","last_undegraded":"2024-04-20T02:54:57.238401+0000","last_fullsized":"2024-04-20T02:54:57.238401+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":218,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:42.084590+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:42.084590+0000","last_peered":"2024-04-20T02:54:42.084590+0000","last_clean":"2024-04-20T02:54:42.084590+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:42.084590+0000","last_undegraded":"2024-04-20T02:54:42.084590+0000","last_fullsized":"2024-04-20T02:54:42.084590+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":246,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-20T02:54:41.161689+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:41.161689+0000","last_peered":"2024-04-20T02:54:41.161689+0000","last_clean":"2024-04-20T02:54:41.161689+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:41.161689+0000","last_undegraded":"2024-04-20T02:54:41.161689+0000","last_fullsized":"2024-04-20T02:54:41.161689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37816,"kb_used_data":2596,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334024,"statfs":{"total":96636764160,"available":96598040576,"internally_reserved":0,"allocated":2658304,"data_stored":1698914,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.749},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.77600000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.73099999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.69299999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.78700000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.80100000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70420,"kb_used_data":34504,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301420,"statfs":{"total":96636764160,"available":96564654080,"internally_reserved":0,"allocated":35332096,"data_stored":36215822,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41699999999999998,"15min":0.39500000000000002},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.71099999999999997,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.59899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.47399999999999998,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.1719999999999997,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.47599999999999998,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.252,"15min":0.252},"max":{"1min":4.2030000000000003,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.626}]},{"osd":3,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.50600000000000001,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.31,"15min":0.31},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.441,"15min":0.435},"min":{"1min":0.30499999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.50800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.26600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.64300000000000002,"5min":1.472,"15min":1.472},"last":0.443},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54600000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45500000000000002,"15min":0.441},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73299999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.48799999999999999},"min":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79200000000000004,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.58799999999999997}]}]},{"osd":2,"up_from":138,"seq":592705486879,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43424,"kb_used_data":15436,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328416,"statfs":{"total":96636764160,"available":96592297984,"internally_reserved":0,"allocated":15806464,"data_stored":15139178,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.44}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.621}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.60699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.68999999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.67400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54720,"kb_used_data":15412,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317120,"statfs":{"total":96636764160,"available":96580730880,"internally_reserved":0,"allocated":15781888,"data_stored":15129550,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.316}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.376}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.438}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.36599999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.502},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.42799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69704,"kb_used_data":34492,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302136,"statfs":{"total":96636764160,"available":96565387264,"internally_reserved":0,"allocated":35319808,"data_stored":36211008,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.627}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83488,"kb_used_data":48276,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288352,"statfs":{"total":96636764160,"available":96551272448,"internally_reserved":0,"allocated":49434624,"data_stored":51702580,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.55600000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.5},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59199999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.75800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.83299999999999996}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.66100000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.746}]}]},{"osd":0,"up_from":15,"seq":64424509499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77704,"kb_used_data":38396,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294136,"statfs":{"total":96636764160,"available":96557195264,"internally_reserved":0,"allocated":39317504,"data_stored":40667054,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.443}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.433}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.626}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.67100000000000004}]}]},{"osd":4,"up_from":167,"seq":717259538457,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48788,"kb_used_data":20864,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323052,"statfs":{"total":96636764160,"available":96586805248,"internally_reserved":0,"allocated":21364736,"data_stored":21150782,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.501}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.44500000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.38}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:04.946 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:05.288 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:05.290 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:05.301 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-04-20T02:55:03.500865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486200,"kb_used_data":210108,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488520,"statfs":{"total":773094113280,"available":772596244480,"internally_reserved":0,"allocated":215150592,"data_stored":217967842,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001209"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251179+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:58.251179+0000","last_peered":"2024-04-20T02:54:58.251179+0000","last_clean":"2024-04-20T02:54:58.251179+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:58.251179+0000","last_undegraded":"2024-04-20T02:54:58.251179+0000","last_fullsized":"2024-04-20T02:54:58.251179+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":277,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251371+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:58.251371+0000","last_peered":"2024-04-20T02:54:58.251371+0000","last_clean":"2024-04-20T02:54:58.251371+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:58.251371+0000","last_undegraded":"2024-04-20T02:54:58.251371+0000","last_fullsized":"2024-04-20T02:54:58.251371+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":276,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.386675+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:58.386675+0000","last_peered":"2024-04-20T02:54:58.386675+0000","last_clean":"2024-04-20T02:54:58.386675+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:58.386675+0000","last_undegraded":"2024-04-20T02:54:58.386675+0000","last_fullsized":"2024-04-20T02:54:58.386675+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":669,"reported_epoch":232,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:57.286187+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:57.286187+0000","last_peered":"2024-04-20T02:54:57.286187+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:57.286187+0000","last_undegraded":"2024-04-20T02:54:57.286187+0000","last_fullsized":"2024-04-20T02:54:57.286187+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251113+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:58.251113+0000","last_peered":"2024-04-20T02:54:58.251113+0000","last_clean":"2024-04-20T02:54:58.251113+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:58.251113+0000","last_undegraded":"2024-04-20T02:54:58.251113+0000","last_fullsized":"2024-04-20T02:54:58.251113+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":661,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.250571+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:58.250571+0000","last_peered":"2024-04-20T02:54:58.250571+0000","last_clean":"2024-04-20T02:54:58.250571+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:58.250571+0000","last_undegraded":"2024-04-20T02:54:58.250571+0000","last_fullsized":"2024-04-20T02:54:58.250571+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":412,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.386696+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:58.386696+0000","last_peered":"2024-04-20T02:54:58.386696+0000","last_clean":"2024-04-20T02:54:58.386696+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:58.386696+0000","last_undegraded":"2024-04-20T02:54:58.386696+0000","last_fullsized":"2024-04-20T02:54:58.386696+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":253,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651192+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:58.651192+0000","last_peered":"2024-04-20T02:54:58.651192+0000","last_clean":"2024-04-20T02:54:58.651192+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:58.651192+0000","last_undegraded":"2024-04-20T02:54:58.651192+0000","last_fullsized":"2024-04-20T02:54:58.651192+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":672,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251321+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:58.251321+0000","last_peered":"2024-04-20T02:54:58.251321+0000","last_clean":"2024-04-20T02:54:58.251321+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:58.251321+0000","last_undegraded":"2024-04-20T02:54:58.251321+0000","last_fullsized":"2024-04-20T02:54:58.251321+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":43,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.278468+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:54:58.278468+0000","last_peered":"2024-04-20T02:54:58.278468+0000","last_clean":"2024-04-20T02:54:58.278468+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:54:58.278468+0000","last_undegraded":"2024-04-20T02:54:58.278468+0000","last_fullsized":"2024-04-20T02:54:58.278468+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":684,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-04-20T02:54:57.238401+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:54:57.238401+0000","last_peered":"2024-04-20T02:54:57.238401+0000","last_clean":"2024-04-20T02:54:57.238401+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:54:57.238401+0000","last_undegraded":"2024-04-20T02:54:57.238401+0000","last_fullsized":"2024-04-20T02:54:57.238401+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":220,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651206+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:58.651206+0000","last_peered":"2024-04-20T02:54:58.651206+0000","last_clean":"2024-04-20T02:54:58.651206+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:58.651206+0000","last_undegraded":"2024-04-20T02:54:58.651206+0000","last_fullsized":"2024-04-20T02:54:58.651206+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":248,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251320+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:58.251320+0000","last_peered":"2024-04-20T02:54:58.251320+0000","last_clean":"2024-04-20T02:54:58.251320+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:58.251320+0000","last_undegraded":"2024-04-20T02:54:58.251320+0000","last_fullsized":"2024-04-20T02:54:58.251320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37816,"kb_used_data":2596,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334024,"statfs":{"total":96636764160,"available":96598040576,"internally_reserved":0,"allocated":2658304,"data_stored":1698914,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.749},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.77600000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.73099999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.69299999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.78700000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.80100000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757300,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70448,"kb_used_data":34528,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301392,"statfs":{"total":96636764160,"available":96564625408,"internally_reserved":0,"allocated":35356672,"data_stored":36225450,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.65700000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.626},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.72299999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.75800000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.745},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69299999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486879,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43424,"kb_used_data":15436,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328416,"statfs":{"total":96636764160,"available":96592297984,"internally_reserved":0,"allocated":15806464,"data_stored":15139178,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.44}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.621}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.60699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.68999999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.67400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54756,"kb_used_data":15448,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317084,"statfs":{"total":96636764160,"available":96580694016,"internally_reserved":0,"allocated":15818752,"data_stored":15143992,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.23499999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.29699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.44}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.626},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.52400000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.47799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574910,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69740,"kb_used_data":34528,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302100,"statfs":{"total":96636764160,"available":96565350400,"internally_reserved":0,"allocated":35356672,"data_stored":36225450,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.48199999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.36199999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.379}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.63800000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.73599999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.58299999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83524,"kb_used_data":48312,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288316,"statfs":{"total":96636764160,"available":96551235584,"internally_reserved":0,"allocated":49471488,"data_stored":51717022,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.80700000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.749}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.76800000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.85699999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61799999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.56799999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.69499999999999995}]}]},{"osd":0,"up_from":15,"seq":64424509499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77704,"kb_used_data":38396,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294136,"statfs":{"total":96636764160,"available":96557195264,"internally_reserved":0,"allocated":39317504,"data_stored":40667054,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.443}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.433}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.626}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.67100000000000004}]}]},{"osd":4,"up_from":167,"seq":717259538457,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48788,"kb_used_data":20864,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323052,"statfs":{"total":96636764160,"available":96586805248,"internally_reserved":0,"allocated":21364736,"data_stored":21150782,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.40699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.501}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.44500000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.38}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:05.303 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:05.631 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:05.632 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:05.644 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-04-20T02:55:05.501419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486240,"kb_used_data":210168,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488480,"statfs":{"total":773094113280,"available":772596203520,"internally_reserved":0,"allocated":215212032,"data_stored":217991912,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.251579"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251179+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:58.251179+0000","last_peered":"2024-04-20T02:54:58.251179+0000","last_clean":"2024-04-20T02:54:58.251179+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:58.251179+0000","last_undegraded":"2024-04-20T02:54:58.251179+0000","last_fullsized":"2024-04-20T02:54:58.251179+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":277,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251371+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:58.251371+0000","last_peered":"2024-04-20T02:54:58.251371+0000","last_clean":"2024-04-20T02:54:58.251371+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:58.251371+0000","last_undegraded":"2024-04-20T02:54:58.251371+0000","last_fullsized":"2024-04-20T02:54:58.251371+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":277,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651385+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:58.651385+0000","last_peered":"2024-04-20T02:54:58.651385+0000","last_clean":"2024-04-20T02:54:58.651385+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:58.651385+0000","last_undegraded":"2024-04-20T02:54:58.651385+0000","last_fullsized":"2024-04-20T02:54:58.651385+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":670,"reported_epoch":233,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:58.386175+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:58.386175+0000","last_peered":"2024-04-20T02:54:58.386175+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:58.386175+0000","last_undegraded":"2024-04-20T02:54:58.386175+0000","last_fullsized":"2024-04-20T02:54:58.386175+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251113+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:58.251113+0000","last_peered":"2024-04-20T02:54:58.251113+0000","last_clean":"2024-04-20T02:54:58.251113+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:58.251113+0000","last_undegraded":"2024-04-20T02:54:58.251113+0000","last_fullsized":"2024-04-20T02:54:58.251113+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":661,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.250571+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:58.250571+0000","last_peered":"2024-04-20T02:54:58.250571+0000","last_clean":"2024-04-20T02:54:58.250571+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:58.250571+0000","last_undegraded":"2024-04-20T02:54:58.250571+0000","last_fullsized":"2024-04-20T02:54:58.250571+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":413,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651335+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:58.651335+0000","last_peered":"2024-04-20T02:54:58.651335+0000","last_clean":"2024-04-20T02:54:58.651335+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:58.651335+0000","last_undegraded":"2024-04-20T02:54:58.651335+0000","last_fullsized":"2024-04-20T02:54:58.651335+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":253,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651192+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:58.651192+0000","last_peered":"2024-04-20T02:54:58.651192+0000","last_clean":"2024-04-20T02:54:58.651192+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:58.651192+0000","last_undegraded":"2024-04-20T02:54:58.651192+0000","last_fullsized":"2024-04-20T02:54:58.651192+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":672,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251321+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:58.251321+0000","last_peered":"2024-04-20T02:54:58.251321+0000","last_clean":"2024-04-20T02:54:58.251321+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:58.251321+0000","last_undegraded":"2024-04-20T02:54:58.251321+0000","last_fullsized":"2024-04-20T02:54:58.251321+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:02.978867+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:02.978867+0000","last_peered":"2024-04-20T02:55:02.978867+0000","last_clean":"2024-04-20T02:55:02.978867+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:02.978867+0000","last_undegraded":"2024-04-20T02:55:02.978867+0000","last_fullsized":"2024-04-20T02:55:02.978867+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":686,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:02.576539+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:02.576539+0000","last_peered":"2024-04-20T02:55:02.576539+0000","last_clean":"2024-04-20T02:55:02.576539+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:02.576539+0000","last_undegraded":"2024-04-20T02:55:02.576539+0000","last_fullsized":"2024-04-20T02:55:02.576539+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":220,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651206+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:58.651206+0000","last_peered":"2024-04-20T02:54:58.651206+0000","last_clean":"2024-04-20T02:54:58.651206+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:58.651206+0000","last_undegraded":"2024-04-20T02:54:58.651206+0000","last_fullsized":"2024-04-20T02:54:58.651206+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":248,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251320+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:58.251320+0000","last_peered":"2024-04-20T02:54:58.251320+0000","last_clean":"2024-04-20T02:54:58.251320+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:58.251320+0000","last_undegraded":"2024-04-20T02:54:58.251320+0000","last_fullsized":"2024-04-20T02:54:58.251320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509504,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2670592,"data_stored":1703728,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.71599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.504}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.438}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.62}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.57599999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757300,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70448,"kb_used_data":34528,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301392,"statfs":{"total":96636764160,"available":96564625408,"internally_reserved":0,"allocated":35356672,"data_stored":36225450,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.65700000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.626},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.72299999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.75800000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.745},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69299999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486880,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43428,"kb_used_data":15448,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328412,"statfs":{"total":96636764160,"available":96592293888,"internally_reserved":0,"allocated":15818752,"data_stored":15143992,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.432},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.84899999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.76800000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.86399999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.82499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54756,"kb_used_data":15448,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317084,"statfs":{"total":96636764160,"available":96580694016,"internally_reserved":0,"allocated":15818752,"data_stored":15143992,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.23499999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.29699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.44}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.626},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.52400000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.47799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574910,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69740,"kb_used_data":34528,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302100,"statfs":{"total":96636764160,"available":96565350400,"internally_reserved":0,"allocated":35356672,"data_stored":36225450,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.48199999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.36199999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.379}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.63800000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.73599999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.58299999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83524,"kb_used_data":48312,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288316,"statfs":{"total":96636764160,"available":96551235584,"internally_reserved":0,"allocated":49471488,"data_stored":51717022,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.80700000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.749}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.76800000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.85699999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61799999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.56799999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.69499999999999995}]}]},{"osd":0,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77716,"kb_used_data":38408,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294124,"statfs":{"total":96636764160,"available":96557182976,"internally_reserved":0,"allocated":39329792,"data_stored":40671868,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.501}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.67400000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.872}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.85699999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.75},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.91200000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538458,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48808,"kb_used_data":20888,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323032,"statfs":{"total":96636764160,"available":96586784768,"internally_reserved":0,"allocated":21389312,"data_stored":21160410,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.375}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59199999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:05.646 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:55:05.646 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 670, 'reported_epoch': 233, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:58.386175+0000', 'last_change': '2024-04-20T02:54:42.178857+0000', 'last_active': '2024-04-20T02:54:58.386175+0000', 'last_peered': '2024-04-20T02:54:58.386175+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:58.386175+0000', 'last_undegraded': '2024-04-20T02:54:58.386175+0000', 'last_fullsized': '2024-04-20T02:54:58.386175+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:55:05.646 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:55:06.053 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:06.053 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":298,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":235,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497868800,"bytes_avail":772596244480,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-20T02:54:05.487796+0000","services":{}},"progress_events":{}} 2024-04-20T02:55:06.067 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:55:06.067 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:06.395 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:06.396 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:06.408 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-04-20T02:55:06.162736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486240,"kb_used_data":210168,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488480,"statfs":{"total":773094113280,"available":772596203520,"internally_reserved":0,"allocated":215212032,"data_stored":217991912,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.662698"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251179+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:54:58.251179+0000","last_peered":"2024-04-20T02:54:58.251179+0000","last_clean":"2024-04-20T02:54:58.251179+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:54:58.251179+0000","last_undegraded":"2024-04-20T02:54:58.251179+0000","last_fullsized":"2024-04-20T02:54:58.251179+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":277,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251371+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:54:58.251371+0000","last_peered":"2024-04-20T02:54:58.251371+0000","last_clean":"2024-04-20T02:54:58.251371+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:54:58.251371+0000","last_undegraded":"2024-04-20T02:54:58.251371+0000","last_fullsized":"2024-04-20T02:54:58.251371+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":277,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651385+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:58.651385+0000","last_peered":"2024-04-20T02:54:58.651385+0000","last_clean":"2024-04-20T02:54:58.651385+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:58.651385+0000","last_undegraded":"2024-04-20T02:54:58.651385+0000","last_fullsized":"2024-04-20T02:54:58.651385+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":670,"reported_epoch":233,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:58.386175+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:58.386175+0000","last_peered":"2024-04-20T02:54:58.386175+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:58.386175+0000","last_undegraded":"2024-04-20T02:54:58.386175+0000","last_fullsized":"2024-04-20T02:54:58.386175+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251113+0000","last_change":"2024-04-20T02:53:56.572271+0000","last_active":"2024-04-20T02:54:58.251113+0000","last_peered":"2024-04-20T02:54:58.251113+0000","last_clean":"2024-04-20T02:54:58.251113+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:54:58.251113+0000","last_undegraded":"2024-04-20T02:54:58.251113+0000","last_fullsized":"2024-04-20T02:54:58.251113+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:44.734595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:44.734595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:04.857537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":661,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.250571+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:54:58.250571+0000","last_peered":"2024-04-20T02:54:58.250571+0000","last_clean":"2024-04-20T02:54:58.250571+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:54:58.250571+0000","last_undegraded":"2024-04-20T02:54:58.250571+0000","last_fullsized":"2024-04-20T02:54:58.250571+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":413,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651335+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:58.651335+0000","last_peered":"2024-04-20T02:54:58.651335+0000","last_clean":"2024-04-20T02:54:58.651335+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:58.651335+0000","last_undegraded":"2024-04-20T02:54:58.651335+0000","last_fullsized":"2024-04-20T02:54:58.651335+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":253,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651192+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:54:58.651192+0000","last_peered":"2024-04-20T02:54:58.651192+0000","last_clean":"2024-04-20T02:54:58.651192+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:54:58.651192+0000","last_undegraded":"2024-04-20T02:54:58.651192+0000","last_fullsized":"2024-04-20T02:54:58.651192+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":672,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251321+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:54:58.251321+0000","last_peered":"2024-04-20T02:54:58.251321+0000","last_clean":"2024-04-20T02:54:58.251321+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:54:58.251321+0000","last_undegraded":"2024-04-20T02:54:58.251321+0000","last_fullsized":"2024-04-20T02:54:58.251321+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:02.978867+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:02.978867+0000","last_peered":"2024-04-20T02:55:02.978867+0000","last_clean":"2024-04-20T02:55:02.978867+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:02.978867+0000","last_undegraded":"2024-04-20T02:55:02.978867+0000","last_fullsized":"2024-04-20T02:55:02.978867+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":686,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:02.576539+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:02.576539+0000","last_peered":"2024-04-20T02:55:02.576539+0000","last_clean":"2024-04-20T02:55:02.576539+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:02.576539+0000","last_undegraded":"2024-04-20T02:55:02.576539+0000","last_fullsized":"2024-04-20T02:55:02.576539+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":220,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651206+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:54:58.651206+0000","last_peered":"2024-04-20T02:54:58.651206+0000","last_clean":"2024-04-20T02:54:58.651206+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:54:58.651206+0000","last_undegraded":"2024-04-20T02:54:58.651206+0000","last_fullsized":"2024-04-20T02:54:58.651206+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":248,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.251320+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:54:58.251320+0000","last_peered":"2024-04-20T02:54:58.251320+0000","last_clean":"2024-04-20T02:54:58.251320+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:54:58.251320+0000","last_undegraded":"2024-04-20T02:54:58.251320+0000","last_fullsized":"2024-04-20T02:54:58.251320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509504,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2670592,"data_stored":1703728,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.71599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.504}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.438}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.62}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.57599999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757300,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70448,"kb_used_data":34528,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301392,"statfs":{"total":96636764160,"available":96564625408,"internally_reserved":0,"allocated":35356672,"data_stored":36225450,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.65700000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.626},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.72299999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.75800000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.745},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69299999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486880,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43428,"kb_used_data":15448,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328412,"statfs":{"total":96636764160,"available":96592293888,"internally_reserved":0,"allocated":15818752,"data_stored":15143992,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.432},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.84899999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.76800000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.86399999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.82499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54756,"kb_used_data":15448,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317084,"statfs":{"total":96636764160,"available":96580694016,"internally_reserved":0,"allocated":15818752,"data_stored":15143992,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.23499999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.29699999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.44}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.56899999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70299999999999996,"5min":1.097,"15min":1.097},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0449999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.626},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.52400000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.47799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574910,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69740,"kb_used_data":34528,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302100,"statfs":{"total":96636764160,"available":96565350400,"internally_reserved":0,"allocated":35356672,"data_stored":36225450,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.48199999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.36199999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.379}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.63800000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.36899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68400000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.73599999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.58299999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83524,"kb_used_data":48312,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288316,"statfs":{"total":96636764160,"available":96551235584,"internally_reserved":0,"allocated":49471488,"data_stored":51717022,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.80700000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.749}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.76800000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.85699999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61799999999999999}]},{"osd":5,"last update":"Sat Apr 20 02:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.439,"15min":0.44600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.50700000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83599999999999997,"5min":1.038,"15min":1.038},"last":0.56799999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.69499999999999995}]}]},{"osd":0,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77716,"kb_used_data":38408,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294124,"statfs":{"total":96636764160,"available":96557182976,"internally_reserved":0,"allocated":39329792,"data_stored":40671868,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.501}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.67400000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.872}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.85699999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.75},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.91200000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538458,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48808,"kb_used_data":20888,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323032,"statfs":{"total":96636764160,"available":96586784768,"internally_reserved":0,"allocated":21389312,"data_stored":21160410,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.375}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59199999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:09.411 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:09.752 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:09.753 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:09.765 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-04-20T02:55:09.502275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486308,"kb_used_data":210240,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488412,"statfs":{"total":773094113280,"available":772596133888,"internally_reserved":0,"allocated":215285760,"data_stored":218020696,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001817"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.551141+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:55:04.551141+0000","last_peered":"2024-04-20T02:55:04.551141+0000","last_clean":"2024-04-20T02:55:04.551141+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:55:04.551141+0000","last_undegraded":"2024-04-20T02:55:04.551141+0000","last_fullsized":"2024-04-20T02:55:04.551141+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":278,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:01.396782+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:55:01.396782+0000","last_peered":"2024-04-20T02:55:01.396782+0000","last_clean":"2024-04-20T02:55:01.396782+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:55:01.396782+0000","last_undegraded":"2024-04-20T02:55:01.396782+0000","last_fullsized":"2024-04-20T02:55:01.396782+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":277,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651385+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:58.651385+0000","last_peered":"2024-04-20T02:54:58.651385+0000","last_clean":"2024-04-20T02:54:58.651385+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:58.651385+0000","last_undegraded":"2024-04-20T02:54:58.651385+0000","last_fullsized":"2024-04-20T02:54:58.651385+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":670,"reported_epoch":233,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:58.386175+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:58.386175+0000","last_peered":"2024-04-20T02:54:58.386175+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:58.386175+0000","last_undegraded":"2024-04-20T02:54:58.386175+0000","last_fullsized":"2024-04-20T02:54:58.386175+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:05.032980+0000","last_change":"2024-04-20T02:55:05.032980+0000","last_active":"2024-04-20T02:55:05.032980+0000","last_peered":"2024-04-20T02:55:05.032980+0000","last_clean":"2024-04-20T02:55:05.032980+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:55:05.032980+0000","last_undegraded":"2024-04-20T02:55:05.032980+0000","last_fullsized":"2024-04-20T02:55:05.032980+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:05.032853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:55:05.032853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:06.092856+0000","scrub_duration":0.0016237370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":663,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273290+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:55:03.273290+0000","last_peered":"2024-04-20T02:55:03.273290+0000","last_clean":"2024-04-20T02:55:03.273290+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:55:03.273290+0000","last_undegraded":"2024-04-20T02:55:03.273290+0000","last_fullsized":"2024-04-20T02:55:03.273290+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":413,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651335+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:58.651335+0000","last_peered":"2024-04-20T02:54:58.651335+0000","last_clean":"2024-04-20T02:54:58.651335+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:58.651335+0000","last_undegraded":"2024-04-20T02:54:58.651335+0000","last_fullsized":"2024-04-20T02:54:58.651335+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":255,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.864996+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:55:04.864996+0000","last_peered":"2024-04-20T02:55:04.864996+0000","last_clean":"2024-04-20T02:55:04.864996+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:55:04.864996+0000","last_undegraded":"2024-04-20T02:55:04.864996+0000","last_fullsized":"2024-04-20T02:55:04.864996+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":673,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:01.396773+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:55:01.396773+0000","last_peered":"2024-04-20T02:55:01.396773+0000","last_clean":"2024-04-20T02:55:01.396773+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:55:01.396773+0000","last_undegraded":"2024-04-20T02:55:01.396773+0000","last_fullsized":"2024-04-20T02:55:01.396773+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:02.978867+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:02.978867+0000","last_peered":"2024-04-20T02:55:02.978867+0000","last_clean":"2024-04-20T02:55:02.978867+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:02.978867+0000","last_undegraded":"2024-04-20T02:55:02.978867+0000","last_fullsized":"2024-04-20T02:55:02.978867+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":686,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:02.576539+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:02.576539+0000","last_peered":"2024-04-20T02:55:02.576539+0000","last_clean":"2024-04-20T02:55:02.576539+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:02.576539+0000","last_undegraded":"2024-04-20T02:55:02.576539+0000","last_fullsized":"2024-04-20T02:55:02.576539+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":222,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.864933+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:55:04.864933+0000","last_peered":"2024-04-20T02:55:04.864933+0000","last_clean":"2024-04-20T02:55:04.864933+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:55:04.864933+0000","last_undegraded":"2024-04-20T02:55:04.864933+0000","last_fullsized":"2024-04-20T02:55:04.864933+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":249,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:01.396791+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:55:01.396791+0000","last_peered":"2024-04-20T02:55:01.396791+0000","last_clean":"2024-04-20T02:55:01.396791+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:55:01.396791+0000","last_undegraded":"2024-04-20T02:55:01.396791+0000","last_fullsized":"2024-04-20T02:55:01.396791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509504,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2670592,"data_stored":1703728,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.71599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.504}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.438}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.62}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.57599999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70472,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301368,"statfs":{"total":96636764160,"available":96564600832,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.93100000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":1.02}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.98199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.71899999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.501},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.745},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69299999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486880,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43428,"kb_used_data":15448,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328412,"statfs":{"total":96636764160,"available":96592293888,"internally_reserved":0,"allocated":15818752,"data_stored":15143992,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.432},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.84899999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.76800000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.86399999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.82499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54768,"kb_used_data":15460,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317072,"statfs":{"total":96636764160,"available":96580681728,"internally_reserved":0,"allocated":15831040,"data_stored":15148806,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.56599999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.66100000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.39200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72499999999999998,"5min":1.097,"15min":1.097},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.67900000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.623}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69764,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302076,"statfs":{"total":96636764160,"available":96565325824,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.316},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.41199999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.39700000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.34300000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67900000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.47699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56999999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.502}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53300000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83532,"kb_used_data":48324,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288308,"statfs":{"total":96636764160,"available":96551227392,"internally_reserved":0,"allocated":49483776,"data_stored":51721836,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.50700000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.60999999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47999999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.66600000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.58199999999999996}]}]},{"osd":0,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77716,"kb_used_data":38408,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294124,"statfs":{"total":96636764160,"available":96557182976,"internally_reserved":0,"allocated":39329792,"data_stored":40671868,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.501}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.67400000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.872}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.85699999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.75},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.91200000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538458,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48808,"kb_used_data":20888,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323032,"statfs":{"total":96636764160,"available":96586784768,"internally_reserved":0,"allocated":21389312,"data_stored":21160410,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.375}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59199999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:09.767 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:10.095 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:10.097 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:10.108 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-04-20T02:55:09.502275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486308,"kb_used_data":210240,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488412,"statfs":{"total":773094113280,"available":772596133888,"internally_reserved":0,"allocated":215285760,"data_stored":218020696,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001817"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.551141+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:55:04.551141+0000","last_peered":"2024-04-20T02:55:04.551141+0000","last_clean":"2024-04-20T02:55:04.551141+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:55:04.551141+0000","last_undegraded":"2024-04-20T02:55:04.551141+0000","last_fullsized":"2024-04-20T02:55:04.551141+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":278,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:01.396782+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:55:01.396782+0000","last_peered":"2024-04-20T02:55:01.396782+0000","last_clean":"2024-04-20T02:55:01.396782+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:55:01.396782+0000","last_undegraded":"2024-04-20T02:55:01.396782+0000","last_fullsized":"2024-04-20T02:55:01.396782+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":277,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651385+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:58.651385+0000","last_peered":"2024-04-20T02:54:58.651385+0000","last_clean":"2024-04-20T02:54:58.651385+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:58.651385+0000","last_undegraded":"2024-04-20T02:54:58.651385+0000","last_fullsized":"2024-04-20T02:54:58.651385+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":670,"reported_epoch":233,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:58.386175+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:58.386175+0000","last_peered":"2024-04-20T02:54:58.386175+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:58.386175+0000","last_undegraded":"2024-04-20T02:54:58.386175+0000","last_fullsized":"2024-04-20T02:54:58.386175+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:05.032980+0000","last_change":"2024-04-20T02:55:05.032980+0000","last_active":"2024-04-20T02:55:05.032980+0000","last_peered":"2024-04-20T02:55:05.032980+0000","last_clean":"2024-04-20T02:55:05.032980+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:55:05.032980+0000","last_undegraded":"2024-04-20T02:55:05.032980+0000","last_fullsized":"2024-04-20T02:55:05.032980+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:05.032853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:55:05.032853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:06.092856+0000","scrub_duration":0.0016237370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":663,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273290+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:55:03.273290+0000","last_peered":"2024-04-20T02:55:03.273290+0000","last_clean":"2024-04-20T02:55:03.273290+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:55:03.273290+0000","last_undegraded":"2024-04-20T02:55:03.273290+0000","last_fullsized":"2024-04-20T02:55:03.273290+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":413,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651335+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:58.651335+0000","last_peered":"2024-04-20T02:54:58.651335+0000","last_clean":"2024-04-20T02:54:58.651335+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:58.651335+0000","last_undegraded":"2024-04-20T02:54:58.651335+0000","last_fullsized":"2024-04-20T02:54:58.651335+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":255,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.864996+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:55:04.864996+0000","last_peered":"2024-04-20T02:55:04.864996+0000","last_clean":"2024-04-20T02:55:04.864996+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:55:04.864996+0000","last_undegraded":"2024-04-20T02:55:04.864996+0000","last_fullsized":"2024-04-20T02:55:04.864996+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":673,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:01.396773+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:55:01.396773+0000","last_peered":"2024-04-20T02:55:01.396773+0000","last_clean":"2024-04-20T02:55:01.396773+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:55:01.396773+0000","last_undegraded":"2024-04-20T02:55:01.396773+0000","last_fullsized":"2024-04-20T02:55:01.396773+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:02.978867+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:02.978867+0000","last_peered":"2024-04-20T02:55:02.978867+0000","last_clean":"2024-04-20T02:55:02.978867+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:02.978867+0000","last_undegraded":"2024-04-20T02:55:02.978867+0000","last_fullsized":"2024-04-20T02:55:02.978867+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":686,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:02.576539+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:02.576539+0000","last_peered":"2024-04-20T02:55:02.576539+0000","last_clean":"2024-04-20T02:55:02.576539+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:02.576539+0000","last_undegraded":"2024-04-20T02:55:02.576539+0000","last_fullsized":"2024-04-20T02:55:02.576539+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":222,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.864933+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:55:04.864933+0000","last_peered":"2024-04-20T02:55:04.864933+0000","last_clean":"2024-04-20T02:55:04.864933+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:55:04.864933+0000","last_undegraded":"2024-04-20T02:55:04.864933+0000","last_fullsized":"2024-04-20T02:55:04.864933+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":249,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:01.396791+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:55:01.396791+0000","last_peered":"2024-04-20T02:55:01.396791+0000","last_clean":"2024-04-20T02:55:01.396791+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:55:01.396791+0000","last_undegraded":"2024-04-20T02:55:01.396791+0000","last_fullsized":"2024-04-20T02:55:01.396791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509504,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2670592,"data_stored":1703728,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.71599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.504}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.438}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.62}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.57599999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70472,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301368,"statfs":{"total":96636764160,"available":96564600832,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.93100000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":1.02}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.98199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.71899999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.501},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.745},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69299999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486880,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43428,"kb_used_data":15448,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328412,"statfs":{"total":96636764160,"available":96592293888,"internally_reserved":0,"allocated":15818752,"data_stored":15143992,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.432},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.84899999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.76800000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.86399999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.82499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54768,"kb_used_data":15460,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317072,"statfs":{"total":96636764160,"available":96580681728,"internally_reserved":0,"allocated":15831040,"data_stored":15148806,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.56599999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.66100000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.39200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72499999999999998,"5min":1.097,"15min":1.097},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.67900000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.623}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69764,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302076,"statfs":{"total":96636764160,"available":96565325824,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.316},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.41199999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.39700000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.34300000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67900000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.47699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56999999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.502}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53300000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83532,"kb_used_data":48324,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288308,"statfs":{"total":96636764160,"available":96551227392,"internally_reserved":0,"allocated":49483776,"data_stored":51721836,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.50700000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.60999999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47999999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.66600000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.58199999999999996}]}]},{"osd":0,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77716,"kb_used_data":38408,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294124,"statfs":{"total":96636764160,"available":96557182976,"internally_reserved":0,"allocated":39329792,"data_stored":40671868,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.501}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.67400000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.872}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.85699999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.75},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.91200000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538458,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48808,"kb_used_data":20888,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323032,"statfs":{"total":96636764160,"available":96586784768,"internally_reserved":0,"allocated":21389312,"data_stored":21160410,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.375}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59199999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:10.110 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:55:10.110 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 670, 'reported_epoch': 233, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:54:58.386175+0000', 'last_change': '2024-04-20T02:54:42.178857+0000', 'last_active': '2024-04-20T02:54:58.386175+0000', 'last_peered': '2024-04-20T02:54:58.386175+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:54:58.386175+0000', 'last_undegraded': '2024-04-20T02:54:58.386175+0000', 'last_fullsized': '2024-04-20T02:54:58.386175+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:55:10.110 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:55:10.519 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:10.519 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":303,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":236,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":497979392,"bytes_avail":772596133888,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-20T02:55:09.502364+0000","services":{"osd":{"daemons":{"summary":"","5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-20T02:55:10.533 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:55:10.533 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:10.857 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:10.859 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:10.870 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-04-20T02:55:09.502275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486308,"kb_used_data":210240,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488412,"statfs":{"total":773094113280,"available":772596133888,"internally_reserved":0,"allocated":215285760,"data_stored":218020696,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001817"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.551141+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:55:04.551141+0000","last_peered":"2024-04-20T02:55:04.551141+0000","last_clean":"2024-04-20T02:55:04.551141+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:55:04.551141+0000","last_undegraded":"2024-04-20T02:55:04.551141+0000","last_fullsized":"2024-04-20T02:55:04.551141+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":278,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:01.396782+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:55:01.396782+0000","last_peered":"2024-04-20T02:55:01.396782+0000","last_clean":"2024-04-20T02:55:01.396782+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:55:01.396782+0000","last_undegraded":"2024-04-20T02:55:01.396782+0000","last_fullsized":"2024-04-20T02:55:01.396782+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":277,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651385+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:54:58.651385+0000","last_peered":"2024-04-20T02:54:58.651385+0000","last_clean":"2024-04-20T02:54:58.651385+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:54:58.651385+0000","last_undegraded":"2024-04-20T02:54:58.651385+0000","last_fullsized":"2024-04-20T02:54:58.651385+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":670,"reported_epoch":233,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:54:58.386175+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:54:58.386175+0000","last_peered":"2024-04-20T02:54:58.386175+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:54:58.386175+0000","last_undegraded":"2024-04-20T02:54:58.386175+0000","last_fullsized":"2024-04-20T02:54:58.386175+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:05.032980+0000","last_change":"2024-04-20T02:55:05.032980+0000","last_active":"2024-04-20T02:55:05.032980+0000","last_peered":"2024-04-20T02:55:05.032980+0000","last_clean":"2024-04-20T02:55:05.032980+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:55:05.032980+0000","last_undegraded":"2024-04-20T02:55:05.032980+0000","last_fullsized":"2024-04-20T02:55:05.032980+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:05.032853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:55:05.032853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:06.092856+0000","scrub_duration":0.0016237370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":663,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273290+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:55:03.273290+0000","last_peered":"2024-04-20T02:55:03.273290+0000","last_clean":"2024-04-20T02:55:03.273290+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:55:03.273290+0000","last_undegraded":"2024-04-20T02:55:03.273290+0000","last_fullsized":"2024-04-20T02:55:03.273290+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":413,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-04-20T02:54:58.651335+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:54:58.651335+0000","last_peered":"2024-04-20T02:54:58.651335+0000","last_clean":"2024-04-20T02:54:58.651335+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:54:58.651335+0000","last_undegraded":"2024-04-20T02:54:58.651335+0000","last_fullsized":"2024-04-20T02:54:58.651335+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":255,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.864996+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:55:04.864996+0000","last_peered":"2024-04-20T02:55:04.864996+0000","last_clean":"2024-04-20T02:55:04.864996+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:55:04.864996+0000","last_undegraded":"2024-04-20T02:55:04.864996+0000","last_fullsized":"2024-04-20T02:55:04.864996+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":673,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:01.396773+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:55:01.396773+0000","last_peered":"2024-04-20T02:55:01.396773+0000","last_clean":"2024-04-20T02:55:01.396773+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:55:01.396773+0000","last_undegraded":"2024-04-20T02:55:01.396773+0000","last_fullsized":"2024-04-20T02:55:01.396773+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:02.978867+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:02.978867+0000","last_peered":"2024-04-20T02:55:02.978867+0000","last_clean":"2024-04-20T02:55:02.978867+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:02.978867+0000","last_undegraded":"2024-04-20T02:55:02.978867+0000","last_fullsized":"2024-04-20T02:55:02.978867+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":686,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:02.576539+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:02.576539+0000","last_peered":"2024-04-20T02:55:02.576539+0000","last_clean":"2024-04-20T02:55:02.576539+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:02.576539+0000","last_undegraded":"2024-04-20T02:55:02.576539+0000","last_fullsized":"2024-04-20T02:55:02.576539+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":222,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.864933+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:55:04.864933+0000","last_peered":"2024-04-20T02:55:04.864933+0000","last_clean":"2024-04-20T02:55:04.864933+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:55:04.864933+0000","last_undegraded":"2024-04-20T02:55:04.864933+0000","last_fullsized":"2024-04-20T02:55:04.864933+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":249,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-04-20T02:55:01.396791+0000","last_change":"2024-04-20T02:53:42.972564+0000","last_active":"2024-04-20T02:55:01.396791+0000","last_peered":"2024-04-20T02:55:01.396791+0000","last_clean":"2024-04-20T02:55:01.396791+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:55:01.396791+0000","last_undegraded":"2024-04-20T02:55:01.396791+0000","last_fullsized":"2024-04-20T02:55:01.396791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:53:42.972465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:53:42.972465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:07.110160+0000","scrub_duration":0.0012488620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509504,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37820,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334020,"statfs":{"total":96636764160,"available":96598036480,"internally_reserved":0,"allocated":2670592,"data_stored":1703728,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.71599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.504}]},{"osd":4,"last update":"Sat Apr 20 02:54:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.438}]},{"osd":5,"last update":"Sat Apr 20 02:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.45200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70599999999999996,"5min":2.431,"15min":2.431},"last":0.62}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.57599999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70472,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301368,"statfs":{"total":96636764160,"available":96564600832,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.93100000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":1.02}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.98199999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.71899999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.501},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.745},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.69299999999999995}]}]},{"osd":2,"up_from":138,"seq":592705486880,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43428,"kb_used_data":15448,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328412,"statfs":{"total":96636764160,"available":96592293888,"internally_reserved":0,"allocated":15818752,"data_stored":15143992,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.432},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.84899999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.76800000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.86399999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.82499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54768,"kb_used_data":15460,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317072,"statfs":{"total":96636764160,"available":96580681728,"internally_reserved":0,"allocated":15831040,"data_stored":15148806,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36399999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.56599999999999995}]},{"osd":4,"last update":"Sat Apr 20 02:54:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.66100000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.39200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72499999999999998,"5min":1.097,"15min":1.097},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.70599999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.67900000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.623}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69764,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302076,"statfs":{"total":96636764160,"available":96565325824,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.316},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.41199999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.39700000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.34300000000000003}]},{"osd":4,"last update":"Sat Apr 20 02:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.67900000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.47699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56999999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.502}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53300000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83532,"kb_used_data":48324,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288308,"statfs":{"total":96636764160,"available":96551227392,"internally_reserved":0,"allocated":49483776,"data_stored":51721836,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.59599999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.50700000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.60999999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47999999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.66600000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.58199999999999996}]}]},{"osd":0,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77716,"kb_used_data":38408,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294124,"statfs":{"total":96636764160,"available":96557182976,"internally_reserved":0,"allocated":39329792,"data_stored":40671868,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.501}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.67400000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.872}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.85699999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.75},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.91200000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538458,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48808,"kb_used_data":20888,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323032,"statfs":{"total":96636764160,"available":96586784768,"internally_reserved":0,"allocated":21389312,"data_stored":21160410,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.375}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59199999999999997}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.437},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:13.873 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:14.214 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:14.216 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:14.227 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-04-20T02:55:13.503205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486416,"kb_used_data":210348,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488304,"statfs":{"total":773094113280,"available":772596023296,"internally_reserved":0,"allocated":215396352,"data_stored":218063722,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001785"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.551141+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:55:04.551141+0000","last_peered":"2024-04-20T02:55:04.551141+0000","last_clean":"2024-04-20T02:55:04.551141+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:55:04.551141+0000","last_undegraded":"2024-04-20T02:55:04.551141+0000","last_fullsized":"2024-04-20T02:55:04.551141+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":279,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273579+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:55:03.273579+0000","last_peered":"2024-04-20T02:55:03.273579+0000","last_clean":"2024-04-20T02:55:03.273579+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:55:03.273579+0000","last_undegraded":"2024-04-20T02:55:03.273579+0000","last_fullsized":"2024-04-20T02:55:03.273579+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":279,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.897951+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:55:04.897951+0000","last_peered":"2024-04-20T02:55:04.897951+0000","last_clean":"2024-04-20T02:55:04.897951+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:55:04.897951+0000","last_undegraded":"2024-04-20T02:55:04.897951+0000","last_fullsized":"2024-04-20T02:55:04.897951+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":672,"reported_epoch":235,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:55:04.787627+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:55:04.787627+0000","last_peered":"2024-04-20T02:55:04.787627+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:55:04.787627+0000","last_undegraded":"2024-04-20T02:55:04.787627+0000","last_fullsized":"2024-04-20T02:55:04.787627+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:05.032980+0000","last_change":"2024-04-20T02:55:05.032980+0000","last_active":"2024-04-20T02:55:05.032980+0000","last_peered":"2024-04-20T02:55:05.032980+0000","last_clean":"2024-04-20T02:55:05.032980+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:55:05.032980+0000","last_undegraded":"2024-04-20T02:55:05.032980+0000","last_fullsized":"2024-04-20T02:55:05.032980+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:05.032853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:55:05.032853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:06.092856+0000","scrub_duration":0.0016237370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":663,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273290+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:55:03.273290+0000","last_peered":"2024-04-20T02:55:03.273290+0000","last_clean":"2024-04-20T02:55:03.273290+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:55:03.273290+0000","last_undegraded":"2024-04-20T02:55:03.273290+0000","last_fullsized":"2024-04-20T02:55:03.273290+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":415,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.897945+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:55:04.897945+0000","last_peered":"2024-04-20T02:55:04.897945+0000","last_clean":"2024-04-20T02:55:04.897945+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:55:04.897945+0000","last_undegraded":"2024-04-20T02:55:04.897945+0000","last_fullsized":"2024-04-20T02:55:04.897945+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":257,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.309466+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:55:13.309466+0000","last_peered":"2024-04-20T02:55:13.309466+0000","last_clean":"2024-04-20T02:55:13.309466+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:55:13.309466+0000","last_undegraded":"2024-04-20T02:55:13.309466+0000","last_fullsized":"2024-04-20T02:55:13.309466+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":674,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273579+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:55:03.273579+0000","last_peered":"2024-04-20T02:55:03.273579+0000","last_clean":"2024-04-20T02:55:03.273579+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:55:03.273579+0000","last_undegraded":"2024-04-20T02:55:03.273579+0000","last_fullsized":"2024-04-20T02:55:03.273579+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.897408+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:04.897408+0000","last_peered":"2024-04-20T02:55:04.897408+0000","last_clean":"2024-04-20T02:55:04.897408+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:04.897408+0000","last_undegraded":"2024-04-20T02:55:04.897408+0000","last_fullsized":"2024-04-20T02:55:04.897408+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":687,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.271169+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:03.271169+0000","last_peered":"2024-04-20T02:55:03.271169+0000","last_clean":"2024-04-20T02:55:03.271169+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:03.271169+0000","last_undegraded":"2024-04-20T02:55:03.271169+0000","last_fullsized":"2024-04-20T02:55:03.271169+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":224,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.309557+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:55:13.309557+0000","last_peered":"2024-04-20T02:55:13.309557+0000","last_clean":"2024-04-20T02:55:13.309557+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:55:13.309557+0000","last_undegraded":"2024-04-20T02:55:13.309557+0000","last_fullsized":"2024-04-20T02:55:13.309557+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":261,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-20T02:55:07.896344+0000","last_change":"2024-04-20T02:55:07.896464+0000","last_active":"2024-04-20T02:55:07.896344+0000","last_peered":"2024-04-20T02:55:07.896344+0000","last_clean":"2024-04-20T02:55:07.896344+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:55:07.896344+0000","last_undegraded":"2024-04-20T02:55:07.896344+0000","last_fullsized":"2024-04-20T02:55:07.896344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_clean_scrub_stamp":"2024-04-20T02:55:07.896324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:31.352835+0000","scrub_duration":0.002383745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37844,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333996,"statfs":{"total":96636764160,"available":96598011904,"internally_reserved":0,"allocated":2695168,"data_stored":1713306,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.754}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.69499999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.88900000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.67600000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.33900000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47499999999999998,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.55800000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.39000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.54200000000000004},"min":{"1min":0.39600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78700000000000003,"5min":2.431,"15min":2.431},"last":0.53200000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.78900000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70472,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301368,"statfs":{"total":96636764160,"available":96564600832,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56200000000000006}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.434}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.47299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35099999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38900000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.50800000000000001}]}]},{"osd":2,"up_from":138,"seq":592705486881,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43452,"kb_used_data":15472,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328388,"statfs":{"total":96636764160,"available":96592269312,"internally_reserved":0,"allocated":15843328,"data_stored":15153570,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.36499999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.316},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.41899999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.432},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.56399999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.53700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.50700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54780,"kb_used_data":15472,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317060,"statfs":{"total":96636764160,"available":96580669440,"internally_reserved":0,"allocated":15843328,"data_stored":15153570,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36299999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.372,"15min":0.372},"max":{"1min":0.66100000000000003,"5min":0.878,"15min":0.878},"last":0.47799999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.39200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72499999999999998,"5min":1.097,"15min":1.097},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.502}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.63400000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.66200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69764,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302076,"statfs":{"total":96636764160,"available":96565325824,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.34599999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.505,"15min":0.49099999999999999},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.44400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.47699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52600000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83548,"kb_used_data":48336,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288292,"statfs":{"total":96636764160,"available":96551211008,"internally_reserved":0,"allocated":49496064,"data_stored":51726600,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.628}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.47499999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.53700000000000003}]}]},{"osd":0,"up_from":15,"seq":64424509501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77740,"kb_used_data":38432,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294100,"statfs":{"total":96636764160,"available":96557158400,"internally_reserved":0,"allocated":39354368,"data_stored":40681446,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.505},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.57799999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.65700000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538459,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48816,"kb_used_data":20900,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323024,"statfs":{"total":96636764160,"available":96586776576,"internally_reserved":0,"allocated":21401600,"data_stored":21165174,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.499}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53600000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.439},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.41599999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:14.229 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:14.558 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:14.560 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:14.571 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-04-20T02:55:14.310519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486416,"kb_used_data":210348,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488304,"statfs":{"total":773094113280,"available":772596023296,"internally_reserved":0,"allocated":215396352,"data_stored":218063722,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.147783"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.551141+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:55:04.551141+0000","last_peered":"2024-04-20T02:55:04.551141+0000","last_clean":"2024-04-20T02:55:04.551141+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:55:04.551141+0000","last_undegraded":"2024-04-20T02:55:04.551141+0000","last_fullsized":"2024-04-20T02:55:04.551141+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":279,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273579+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:55:03.273579+0000","last_peered":"2024-04-20T02:55:03.273579+0000","last_clean":"2024-04-20T02:55:03.273579+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:55:03.273579+0000","last_undegraded":"2024-04-20T02:55:03.273579+0000","last_fullsized":"2024-04-20T02:55:03.273579+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":279,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.897951+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:55:04.897951+0000","last_peered":"2024-04-20T02:55:04.897951+0000","last_clean":"2024-04-20T02:55:04.897951+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:55:04.897951+0000","last_undegraded":"2024-04-20T02:55:04.897951+0000","last_fullsized":"2024-04-20T02:55:04.897951+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":672,"reported_epoch":235,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:55:04.787627+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:55:04.787627+0000","last_peered":"2024-04-20T02:55:04.787627+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:55:04.787627+0000","last_undegraded":"2024-04-20T02:55:04.787627+0000","last_fullsized":"2024-04-20T02:55:04.787627+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:05.032980+0000","last_change":"2024-04-20T02:55:05.032980+0000","last_active":"2024-04-20T02:55:05.032980+0000","last_peered":"2024-04-20T02:55:05.032980+0000","last_clean":"2024-04-20T02:55:05.032980+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:55:05.032980+0000","last_undegraded":"2024-04-20T02:55:05.032980+0000","last_fullsized":"2024-04-20T02:55:05.032980+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:05.032853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:55:05.032853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:06.092856+0000","scrub_duration":0.0016237370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":663,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273290+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:55:03.273290+0000","last_peered":"2024-04-20T02:55:03.273290+0000","last_clean":"2024-04-20T02:55:03.273290+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:55:03.273290+0000","last_undegraded":"2024-04-20T02:55:03.273290+0000","last_fullsized":"2024-04-20T02:55:03.273290+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":415,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.897945+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:55:04.897945+0000","last_peered":"2024-04-20T02:55:04.897945+0000","last_clean":"2024-04-20T02:55:04.897945+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:55:04.897945+0000","last_undegraded":"2024-04-20T02:55:04.897945+0000","last_fullsized":"2024-04-20T02:55:04.897945+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":257,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.309466+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:55:13.309466+0000","last_peered":"2024-04-20T02:55:13.309466+0000","last_clean":"2024-04-20T02:55:13.309466+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:55:13.309466+0000","last_undegraded":"2024-04-20T02:55:13.309466+0000","last_fullsized":"2024-04-20T02:55:13.309466+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":674,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273579+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:55:03.273579+0000","last_peered":"2024-04-20T02:55:03.273579+0000","last_clean":"2024-04-20T02:55:03.273579+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:55:03.273579+0000","last_undegraded":"2024-04-20T02:55:03.273579+0000","last_fullsized":"2024-04-20T02:55:03.273579+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.897408+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:04.897408+0000","last_peered":"2024-04-20T02:55:04.897408+0000","last_clean":"2024-04-20T02:55:04.897408+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:04.897408+0000","last_undegraded":"2024-04-20T02:55:04.897408+0000","last_fullsized":"2024-04-20T02:55:04.897408+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":687,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.271169+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:03.271169+0000","last_peered":"2024-04-20T02:55:03.271169+0000","last_clean":"2024-04-20T02:55:03.271169+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:03.271169+0000","last_undegraded":"2024-04-20T02:55:03.271169+0000","last_fullsized":"2024-04-20T02:55:03.271169+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":224,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.309557+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:55:13.309557+0000","last_peered":"2024-04-20T02:55:13.309557+0000","last_clean":"2024-04-20T02:55:13.309557+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:55:13.309557+0000","last_undegraded":"2024-04-20T02:55:13.309557+0000","last_fullsized":"2024-04-20T02:55:13.309557+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":261,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-20T02:55:07.896344+0000","last_change":"2024-04-20T02:55:07.896464+0000","last_active":"2024-04-20T02:55:07.896344+0000","last_peered":"2024-04-20T02:55:07.896344+0000","last_clean":"2024-04-20T02:55:07.896344+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:55:07.896344+0000","last_undegraded":"2024-04-20T02:55:07.896344+0000","last_fullsized":"2024-04-20T02:55:07.896344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_clean_scrub_stamp":"2024-04-20T02:55:07.896324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:31.352835+0000","scrub_duration":0.002383745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37844,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333996,"statfs":{"total":96636764160,"available":96598011904,"internally_reserved":0,"allocated":2695168,"data_stored":1713306,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.754}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.69499999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.88900000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.67600000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.33900000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47499999999999998,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.55800000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.39000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.54200000000000004},"min":{"1min":0.39600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78700000000000003,"5min":2.431,"15min":2.431},"last":0.53200000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.78900000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70472,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301368,"statfs":{"total":96636764160,"available":96564600832,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56200000000000006}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.434}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.47299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35099999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38900000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.50800000000000001}]}]},{"osd":2,"up_from":138,"seq":592705486881,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43452,"kb_used_data":15472,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328388,"statfs":{"total":96636764160,"available":96592269312,"internally_reserved":0,"allocated":15843328,"data_stored":15153570,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.36499999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.316},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.41899999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.432},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.56399999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.53700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.50700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54780,"kb_used_data":15472,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317060,"statfs":{"total":96636764160,"available":96580669440,"internally_reserved":0,"allocated":15843328,"data_stored":15153570,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36299999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.372,"15min":0.372},"max":{"1min":0.66100000000000003,"5min":0.878,"15min":0.878},"last":0.47799999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.39200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72499999999999998,"5min":1.097,"15min":1.097},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.502}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.63400000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.66200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69764,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302076,"statfs":{"total":96636764160,"available":96565325824,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.34599999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.505,"15min":0.49099999999999999},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.44400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.47699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52600000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83548,"kb_used_data":48336,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288292,"statfs":{"total":96636764160,"available":96551211008,"internally_reserved":0,"allocated":49496064,"data_stored":51726600,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.628}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.47499999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.53700000000000003}]}]},{"osd":0,"up_from":15,"seq":64424509501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77740,"kb_used_data":38432,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294100,"statfs":{"total":96636764160,"available":96557158400,"internally_reserved":0,"allocated":39354368,"data_stored":40681446,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.505},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.57799999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.65700000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538459,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48816,"kb_used_data":20900,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323024,"statfs":{"total":96636764160,"available":96586776576,"internally_reserved":0,"allocated":21401600,"data_stored":21165174,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.499}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53600000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.439},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.41599999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:14.573 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-04-20T02:55:14.573 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "187'261", 'reported_seq': 672, 'reported_epoch': 235, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-04-20T02:55:04.787627+0000', 'last_change': '2024-04-20T02:54:42.178857+0000', 'last_active': '2024-04-20T02:55:04.787627+0000', 'last_peered': '2024-04-20T02:55:04.787627+0000', 'last_clean': '2024-04-20T02:54:10.355483+0000', 'last_became_active': '2024-04-20T02:54:12.176645+0000', 'last_became_peered': '2024-04-20T02:54:12.176645+0000', 'last_unstale': '2024-04-20T02:55:04.787627+0000', 'last_undegraded': '2024-04-20T02:55:04.787627+0000', 'last_fullsized': '2024-04-20T02:55:04.787627+0000', 'mapping_epoch': 226, 'log_start': "180'258", 'ondisk_log_start': "180'258", 'created': 21, 'last_epoch_clean': 153, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "163'239", 'last_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-20T02:50:33.213730+0000', 'last_clean_scrub_stamp': '2024-04-20T02:53:08.766836+0000', 'objects_scrubbed': 30, 'log_size': 3, 'log_dups_size': 283, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0.006586525, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28008758, 'num_objects': 33, 'num_object_clones': 0, 'num_object_copies': 99, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 33, 'num_objects_unfound': 0, 'num_objects_dirty': 33, 'num_whiteouts': 0, 'num_read': 245, 'num_read_kb': 8242, 'num_write': 315, 'num_write_kb': 78469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 18507811, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4, 3], 'acting': [0, 3, 5], 'avail_no_missing': ['0', '3', '5'], 'object_location_counts': [{'shards': '0,3,5', 'objects': 33}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-20T02:55:14.573 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-20T02:55:14.974 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:14.975 INFO:teuthology.orchestra.run.smithi105.stdout:{"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":307,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":238,"num_osds":8,"num_up_osds":8,"osd_up_since":1713581586,"num_in_osds":8,"osd_in_since":1713581635,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":130,"data_bytes":98666436,"bytes_used":498089984,"bytes_avail":772596023296,"bytes_total":773094113280,"misplaced_objects":33,"misplaced_total":390,"misplaced_ratio":0.08461538461538462},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-04-20T02:55:09.502364+0000","services":{"osd":{"daemons":{"summary":"","5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-20T02:55:14.989 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-20T02:55:14.989 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:15.314 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:15.315 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:15.327 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-04-20T02:55:14.310519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":486416,"kb_used_data":210348,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754488304,"statfs":{"total":773094113280,"available":772596023296,"internally_reserved":0,"allocated":215396352,"data_stored":218063722,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.147783"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.551141+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:55:04.551141+0000","last_peered":"2024-04-20T02:55:04.551141+0000","last_clean":"2024-04-20T02:55:04.551141+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:55:04.551141+0000","last_undegraded":"2024-04-20T02:55:04.551141+0000","last_fullsized":"2024-04-20T02:55:04.551141+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":279,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273579+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:55:03.273579+0000","last_peered":"2024-04-20T02:55:03.273579+0000","last_clean":"2024-04-20T02:55:03.273579+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:55:03.273579+0000","last_undegraded":"2024-04-20T02:55:03.273579+0000","last_fullsized":"2024-04-20T02:55:03.273579+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":279,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.897951+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:55:04.897951+0000","last_peered":"2024-04-20T02:55:04.897951+0000","last_clean":"2024-04-20T02:55:04.897951+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:55:04.897951+0000","last_undegraded":"2024-04-20T02:55:04.897951+0000","last_fullsized":"2024-04-20T02:55:04.897951+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":672,"reported_epoch":235,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-20T02:55:04.787627+0000","last_change":"2024-04-20T02:54:42.178857+0000","last_active":"2024-04-20T02:55:04.787627+0000","last_peered":"2024-04-20T02:55:04.787627+0000","last_clean":"2024-04-20T02:54:10.355483+0000","last_became_active":"2024-04-20T02:54:12.176645+0000","last_became_peered":"2024-04-20T02:54:12.176645+0000","last_unstale":"2024-04-20T02:55:04.787627+0000","last_undegraded":"2024-04-20T02:55:04.787627+0000","last_fullsized":"2024-04-20T02:55:04.787627+0000","mapping_epoch":226,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'239","last_scrub_stamp":"2024-04-20T02:53:08.766836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:53:08.766836+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065865250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18507811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":33}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:05.032980+0000","last_change":"2024-04-20T02:55:05.032980+0000","last_active":"2024-04-20T02:55:05.032980+0000","last_peered":"2024-04-20T02:55:05.032980+0000","last_clean":"2024-04-20T02:55:05.032980+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:55:05.032980+0000","last_undegraded":"2024-04-20T02:55:05.032980+0000","last_fullsized":"2024-04-20T02:55:05.032980+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:05.032853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:55:05.032853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:06.092856+0000","scrub_duration":0.0016237370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":663,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273290+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:55:03.273290+0000","last_peered":"2024-04-20T02:55:03.273290+0000","last_clean":"2024-04-20T02:55:03.273290+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:55:03.273290+0000","last_undegraded":"2024-04-20T02:55:03.273290+0000","last_fullsized":"2024-04-20T02:55:03.273290+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":415,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.897945+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:55:04.897945+0000","last_peered":"2024-04-20T02:55:04.897945+0000","last_clean":"2024-04-20T02:55:04.897945+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:55:04.897945+0000","last_undegraded":"2024-04-20T02:55:04.897945+0000","last_fullsized":"2024-04-20T02:55:04.897945+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":257,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.309466+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:55:13.309466+0000","last_peered":"2024-04-20T02:55:13.309466+0000","last_clean":"2024-04-20T02:55:13.309466+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:55:13.309466+0000","last_undegraded":"2024-04-20T02:55:13.309466+0000","last_fullsized":"2024-04-20T02:55:13.309466+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":674,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273579+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:55:03.273579+0000","last_peered":"2024-04-20T02:55:03.273579+0000","last_clean":"2024-04-20T02:55:03.273579+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:55:03.273579+0000","last_undegraded":"2024-04-20T02:55:03.273579+0000","last_fullsized":"2024-04-20T02:55:03.273579+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.897408+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:04.897408+0000","last_peered":"2024-04-20T02:55:04.897408+0000","last_clean":"2024-04-20T02:55:04.897408+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:04.897408+0000","last_undegraded":"2024-04-20T02:55:04.897408+0000","last_fullsized":"2024-04-20T02:55:04.897408+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":687,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.271169+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:03.271169+0000","last_peered":"2024-04-20T02:55:03.271169+0000","last_clean":"2024-04-20T02:55:03.271169+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:03.271169+0000","last_undegraded":"2024-04-20T02:55:03.271169+0000","last_fullsized":"2024-04-20T02:55:03.271169+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":224,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.309557+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:55:13.309557+0000","last_peered":"2024-04-20T02:55:13.309557+0000","last_clean":"2024-04-20T02:55:13.309557+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:55:13.309557+0000","last_undegraded":"2024-04-20T02:55:13.309557+0000","last_fullsized":"2024-04-20T02:55:13.309557+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":261,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-20T02:55:07.896344+0000","last_change":"2024-04-20T02:55:07.896464+0000","last_active":"2024-04-20T02:55:07.896344+0000","last_peered":"2024-04-20T02:55:07.896344+0000","last_clean":"2024-04-20T02:55:07.896344+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:55:07.896344+0000","last_undegraded":"2024-04-20T02:55:07.896344+0000","last_fullsized":"2024-04-20T02:55:07.896344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_clean_scrub_stamp":"2024-04-20T02:55:07.896324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:31.352835+0000","scrub_duration":0.002383745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":100933456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37844,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333996,"statfs":{"total":96636764160,"available":96598011904,"internally_reserved":0,"allocated":2695168,"data_stored":1713306,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.754}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.69499999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.88900000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.67600000000000005}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.33900000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47499999999999998,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.55800000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.39000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.54200000000000004},"min":{"1min":0.39600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78700000000000003,"5min":2.431,"15min":2.431},"last":0.53200000000000003}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.78900000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70472,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301368,"statfs":{"total":96636764160,"available":96564600832,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56200000000000006}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.434}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.47299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35099999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38900000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.50800000000000001}]}]},{"osd":2,"up_from":138,"seq":592705486881,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43452,"kb_used_data":15472,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328388,"statfs":{"total":96636764160,"available":96592269312,"internally_reserved":0,"allocated":15843328,"data_stored":15153570,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.36499999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.316},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.41899999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.432},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.44800000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.56399999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.53700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.50700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54780,"kb_used_data":15472,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317060,"statfs":{"total":96636764160,"available":96580669440,"internally_reserved":0,"allocated":15843328,"data_stored":15153570,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36299999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.372,"15min":0.372},"max":{"1min":0.66100000000000003,"5min":0.878,"15min":0.878},"last":0.47799999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.39200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72499999999999998,"5min":1.097,"15min":1.097},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.502}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.63400000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.66200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69764,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302076,"statfs":{"total":96636764160,"available":96565325824,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.34599999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.505,"15min":0.49099999999999999},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.44400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.47699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52600000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83548,"kb_used_data":48336,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288292,"statfs":{"total":96636764160,"available":96551211008,"internally_reserved":0,"allocated":49496064,"data_stored":51726600,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.628}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.47499999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.53700000000000003}]}]},{"osd":0,"up_from":15,"seq":64424509501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77740,"kb_used_data":38432,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294100,"statfs":{"total":96636764160,"available":96557158400,"internally_reserved":0,"allocated":39354368,"data_stored":40681446,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.505},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.56000000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.57799999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.65700000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538459,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48816,"kb_used_data":20900,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94323024,"statfs":{"total":96636764160,"available":96586776576,"internally_reserved":0,"allocated":21401600,"data_stored":21165174,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.499}]},{"osd":1,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":6.1210000000000004,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53600000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.439},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.41599999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.46999999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:54:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":18759680,"data_stored":19980796,"data_compressed":1020351,"data_compressed_allocated":1105920,"data_compressed_original":2383872,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:18.329 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:18.666 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:18.668 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:18.680 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-04-20T02:55:17.504157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":119920800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":503236,"kb_used_data":226088,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754471484,"statfs":{"total":773094113280,"available":772578799616,"internally_reserved":0,"allocated":231514112,"data_stored":236243974,"data_compressed":12076841,"data_compressed_allocated":13131776,"data_compressed_original":29200384,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":18987344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001881"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.551141+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:55:04.551141+0000","last_peered":"2024-04-20T02:55:04.551141+0000","last_clean":"2024-04-20T02:55:04.551141+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:55:04.551141+0000","last_undegraded":"2024-04-20T02:55:04.551141+0000","last_fullsized":"2024-04-20T02:55:04.551141+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":279,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273579+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:55:03.273579+0000","last_peered":"2024-04-20T02:55:03.273579+0000","last_clean":"2024-04-20T02:55:03.273579+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:55:03.273579+0000","last_undegraded":"2024-04-20T02:55:03.273579+0000","last_fullsized":"2024-04-20T02:55:03.273579+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":280,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-20T02:55:07.861328+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:55:07.861328+0000","last_peered":"2024-04-20T02:55:07.861328+0000","last_clean":"2024-04-20T02:55:07.861328+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:55:07.861328+0000","last_undegraded":"2024-04-20T02:55:07.861328+0000","last_fullsized":"2024-04-20T02:55:07.861328+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":766,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:14.316425+0000","last_change":"2024-04-20T02:55:14.316425+0000","last_active":"2024-04-20T02:55:14.316425+0000","last_peered":"2024-04-20T02:55:14.316425+0000","last_clean":"2024-04-20T02:55:14.316425+0000","last_became_active":"2024-04-20T02:55:14.316051+0000","last_became_peered":"2024-04-20T02:55:14.316051+0000","last_unstale":"2024-04-20T02:55:14.316425+0000","last_undegraded":"2024-04-20T02:55:14.316425+0000","last_fullsized":"2024-04-20T02:55:14.316425+0000","mapping_epoch":237,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'261","last_scrub_stamp":"2024-04-20T02:55:13.079050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:55:13.079050+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:17.988932+0000","scrub_duration":0.0067537669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":37495155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:05.032980+0000","last_change":"2024-04-20T02:55:05.032980+0000","last_active":"2024-04-20T02:55:05.032980+0000","last_peered":"2024-04-20T02:55:05.032980+0000","last_clean":"2024-04-20T02:55:05.032980+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:55:05.032980+0000","last_undegraded":"2024-04-20T02:55:05.032980+0000","last_fullsized":"2024-04-20T02:55:05.032980+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:05.032853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:55:05.032853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:06.092856+0000","scrub_duration":0.0016237370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":663,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273290+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:55:03.273290+0000","last_peered":"2024-04-20T02:55:03.273290+0000","last_clean":"2024-04-20T02:55:03.273290+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:55:03.273290+0000","last_undegraded":"2024-04-20T02:55:03.273290+0000","last_fullsized":"2024-04-20T02:55:03.273290+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":416,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-20T02:55:07.861410+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:55:07.861410+0000","last_peered":"2024-04-20T02:55:07.861410+0000","last_clean":"2024-04-20T02:55:07.861410+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:55:07.861410+0000","last_undegraded":"2024-04-20T02:55:07.861410+0000","last_fullsized":"2024-04-20T02:55:07.861410+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":257,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.309466+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:55:13.309466+0000","last_peered":"2024-04-20T02:55:13.309466+0000","last_clean":"2024-04-20T02:55:13.309466+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:55:13.309466+0000","last_undegraded":"2024-04-20T02:55:13.309466+0000","last_fullsized":"2024-04-20T02:55:13.309466+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":674,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273579+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:55:03.273579+0000","last_peered":"2024-04-20T02:55:03.273579+0000","last_clean":"2024-04-20T02:55:03.273579+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:55:03.273579+0000","last_undegraded":"2024-04-20T02:55:03.273579+0000","last_fullsized":"2024-04-20T02:55:03.273579+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.543204+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:13.543204+0000","last_peered":"2024-04-20T02:55:13.543204+0000","last_clean":"2024-04-20T02:55:13.543204+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:13.543204+0000","last_undegraded":"2024-04-20T02:55:13.543204+0000","last_fullsized":"2024-04-20T02:55:13.543204+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":689,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.310744+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:13.310744+0000","last_peered":"2024-04-20T02:55:13.310744+0000","last_clean":"2024-04-20T02:55:13.310744+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:13.310744+0000","last_undegraded":"2024-04-20T02:55:13.310744+0000","last_fullsized":"2024-04-20T02:55:13.310744+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":224,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.309557+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:55:13.309557+0000","last_peered":"2024-04-20T02:55:13.309557+0000","last_clean":"2024-04-20T02:55:13.309557+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:55:13.309557+0000","last_undegraded":"2024-04-20T02:55:13.309557+0000","last_fullsized":"2024-04-20T02:55:13.309557+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":261,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-20T02:55:07.896344+0000","last_change":"2024-04-20T02:55:07.896464+0000","last_active":"2024-04-20T02:55:07.896344+0000","last_peered":"2024-04-20T02:55:07.896344+0000","last_clean":"2024-04-20T02:55:07.896344+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:55:07.896344+0000","last_undegraded":"2024-04-20T02:55:07.896344+0000","last_fullsized":"2024-04-20T02:55:07.896344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_clean_scrub_stamp":"2024-04-20T02:55:07.896324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:31.352835+0000","scrub_duration":0.002383745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":119920800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212000768,"data_stored":225989258,"data_compressed":12060887,"data_compressed_allocated":13033472,"data_compressed_original":27664384,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509506,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37844,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333996,"statfs":{"total":96636764160,"available":96598011904,"internally_reserved":0,"allocated":2695168,"data_stored":1713306,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.71399999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.68600000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.65000000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.70299999999999996}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.33900000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47499999999999998,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.45800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.39000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.502},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.54200000000000004},"min":{"1min":0.39600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78700000000000003,"5min":2.431,"15min":2.431},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.55300000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70472,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301368,"statfs":{"total":96636764160,"available":96564600832,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56200000000000006}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.434}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.47299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35099999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38900000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.50800000000000001}]}]},{"osd":2,"up_from":138,"seq":592705486882,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43116,"kb_used_data":14776,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328724,"statfs":{"total":96636764160,"available":96592613376,"internally_reserved":0,"allocated":15130624,"data_stored":14830271,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.38},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.36899999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.40000000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.34399999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.39100000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84899999999999998,"5min":1.462,"15min":1.462},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.60399999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.67200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54780,"kb_used_data":15472,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317060,"statfs":{"total":96636764160,"available":96580669440,"internally_reserved":0,"allocated":15843328,"data_stored":15153570,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36299999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.372,"15min":0.372},"max":{"1min":0.66100000000000003,"5min":0.878,"15min":0.878},"last":0.47799999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.39200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72499999999999998,"5min":1.097,"15min":1.097},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.502}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.63400000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.66200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69764,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302076,"statfs":{"total":96636764160,"available":96565325824,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.34599999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.505,"15min":0.49099999999999999},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.44400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.47699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52600000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83548,"kb_used_data":48336,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288292,"statfs":{"total":96636764160,"available":96551211008,"internally_reserved":0,"allocated":49496064,"data_stored":51726600,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.628}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.47499999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.53700000000000003}]}]},{"osd":0,"up_from":15,"seq":64424509502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77752,"kb_used_data":38084,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294088,"statfs":{"total":96636764160,"available":96557146112,"internally_reserved":0,"allocated":38998016,"data_stored":40520952,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.39300000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.314}]},{"osd":4,"last update":"Sat Apr 20 02:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58099999999999996,"15min":0.57999999999999996},"min":{"1min":0.40500000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.57599999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.238,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.52800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.58099999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.627}]}]},{"osd":4,"up_from":167,"seq":717259538460,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65960,"kb_used_data":37684,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94305880,"statfs":{"total":96636764160,"available":96569221120,"internally_reserved":0,"allocated":38588416,"data_stored":39829219,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.248,"15min":0.248},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72099999999999997,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.626}]},{"osd":1,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.33300000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53000000000000003,"15min":0.51800000000000002},"min":{"1min":0.45600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72599999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.69399999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47499999999999998,"15min":0.47099999999999997},"min":{"1min":0.35999999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65500000000000003,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.42099999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.77200000000000002,"5min":1.085,"15min":1.085},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.41999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.83099999999999996,"15min":0.877},"min":{"1min":0.375,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82899999999999996,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.47399999999999998,"15min":0.46600000000000003},"min":{"1min":0.39700000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69099999999999995,"5min":0.753,"15min":0.753},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.376,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.47399999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.48799999999999999,"15min":0.47599999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.39200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.38400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.503},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.46500000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:18.682 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-20T02:55:18.682 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:19.005 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:19.007 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-04-20T02:55:17.504157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":119920800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":503236,"kb_used_data":226088,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754471484,"statfs":{"total":773094113280,"available":772578799616,"internally_reserved":0,"allocated":231514112,"data_stored":236243974,"data_compressed":12076841,"data_compressed_allocated":13131776,"data_compressed_original":29200384,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":18987344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001881"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:04.551141+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:55:04.551141+0000","last_peered":"2024-04-20T02:55:04.551141+0000","last_clean":"2024-04-20T02:55:04.551141+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:55:04.551141+0000","last_undegraded":"2024-04-20T02:55:04.551141+0000","last_fullsized":"2024-04-20T02:55:04.551141+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":279,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273579+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:55:03.273579+0000","last_peered":"2024-04-20T02:55:03.273579+0000","last_clean":"2024-04-20T02:55:03.273579+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:55:03.273579+0000","last_undegraded":"2024-04-20T02:55:03.273579+0000","last_fullsized":"2024-04-20T02:55:03.273579+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:02.184646+0000","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":280,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-20T02:55:07.861328+0000","last_change":"2024-04-20T02:54:12.775218+0000","last_active":"2024-04-20T02:55:07.861328+0000","last_peered":"2024-04-20T02:55:07.861328+0000","last_clean":"2024-04-20T02:55:07.861328+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:55:07.861328+0000","last_undegraded":"2024-04-20T02:55:07.861328+0000","last_fullsized":"2024-04-20T02:55:07.861328+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.775079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.775079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:16.043902+0000","scrub_duration":0.0089395459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":766,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:14.316425+0000","last_change":"2024-04-20T02:55:14.316425+0000","last_active":"2024-04-20T02:55:14.316425+0000","last_peered":"2024-04-20T02:55:14.316425+0000","last_clean":"2024-04-20T02:55:14.316425+0000","last_became_active":"2024-04-20T02:55:14.316051+0000","last_became_peered":"2024-04-20T02:55:14.316051+0000","last_unstale":"2024-04-20T02:55:14.316425+0000","last_undegraded":"2024-04-20T02:55:14.316425+0000","last_fullsized":"2024-04-20T02:55:14.316425+0000","mapping_epoch":237,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'261","last_scrub_stamp":"2024-04-20T02:55:13.079050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:55:13.079050+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:17.988932+0000","scrub_duration":0.0067537669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":37495155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:05.032980+0000","last_change":"2024-04-20T02:55:05.032980+0000","last_active":"2024-04-20T02:55:05.032980+0000","last_peered":"2024-04-20T02:55:05.032980+0000","last_clean":"2024-04-20T02:55:05.032980+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:55:05.032980+0000","last_undegraded":"2024-04-20T02:55:05.032980+0000","last_fullsized":"2024-04-20T02:55:05.032980+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:05.032853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:55:05.032853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:06.092856+0000","scrub_duration":0.0016237370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":663,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273290+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:55:03.273290+0000","last_peered":"2024-04-20T02:55:03.273290+0000","last_clean":"2024-04-20T02:55:03.273290+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:55:03.273290+0000","last_undegraded":"2024-04-20T02:55:03.273290+0000","last_fullsized":"2024-04-20T02:55:03.273290+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:48.085761+0000","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":416,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-20T02:55:07.861410+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:55:07.861410+0000","last_peered":"2024-04-20T02:55:07.861410+0000","last_clean":"2024-04-20T02:55:07.861410+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:55:07.861410+0000","last_undegraded":"2024-04-20T02:55:07.861410+0000","last_fullsized":"2024-04-20T02:55:07.861410+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":257,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.309466+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:55:13.309466+0000","last_peered":"2024-04-20T02:55:13.309466+0000","last_clean":"2024-04-20T02:55:13.309466+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:55:13.309466+0000","last_undegraded":"2024-04-20T02:55:13.309466+0000","last_fullsized":"2024-04-20T02:55:13.309466+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:24.151835+0000","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":674,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-04-20T02:55:03.273579+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:55:03.273579+0000","last_peered":"2024-04-20T02:55:03.273579+0000","last_clean":"2024-04-20T02:55:03.273579+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:55:03.273579+0000","last_undegraded":"2024-04-20T02:55:03.273579+0000","last_fullsized":"2024-04-20T02:55:03.273579+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:27.855025+0000","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.543204+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:13.543204+0000","last_peered":"2024-04-20T02:55:13.543204+0000","last_clean":"2024-04-20T02:55:13.543204+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:13.543204+0000","last_undegraded":"2024-04-20T02:55:13.543204+0000","last_fullsized":"2024-04-20T02:55:13.543204+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":689,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.310744+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:13.310744+0000","last_peered":"2024-04-20T02:55:13.310744+0000","last_clean":"2024-04-20T02:55:13.310744+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:13.310744+0000","last_undegraded":"2024-04-20T02:55:13.310744+0000","last_fullsized":"2024-04-20T02:55:13.310744+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":224,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.309557+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:55:13.309557+0000","last_peered":"2024-04-20T02:55:13.309557+0000","last_clean":"2024-04-20T02:55:13.309557+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:55:13.309557+0000","last_undegraded":"2024-04-20T02:55:13.309557+0000","last_fullsized":"2024-04-20T02:55:13.309557+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:03.231743+0000","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":261,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-04-20T02:55:07.896344+0000","last_change":"2024-04-20T02:55:07.896464+0000","last_active":"2024-04-20T02:55:07.896344+0000","last_peered":"2024-04-20T02:55:07.896344+0000","last_clean":"2024-04-20T02:55:07.896344+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:55:07.896344+0000","last_undegraded":"2024-04-20T02:55:07.896344+0000","last_fullsized":"2024-04-20T02:55:07.896344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_clean_scrub_stamp":"2024-04-20T02:55:07.896324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:31.352835+0000","scrub_duration":0.002383745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":119920800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212000768,"data_stored":225989258,"data_compressed":12060887,"data_compressed_allocated":13033472,"data_compressed_original":27664384,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509506,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37844,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94333996,"statfs":{"total":96636764160,"available":96598011904,"internally_reserved":0,"allocated":2695168,"data_stored":1713306,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.71399999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.68600000000000005}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.65000000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.70299999999999996}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.33900000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47499999999999998,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.45800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.39000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.502},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.54200000000000004},"min":{"1min":0.39600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78700000000000003,"5min":2.431,"15min":2.431},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.55300000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70472,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94301368,"statfs":{"total":96636764160,"available":96564600832,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.56200000000000006}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.434}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.47299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35099999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38900000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.46700000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.50800000000000001}]}]},{"osd":2,"up_from":138,"seq":592705486882,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43116,"kb_used_data":14776,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328724,"statfs":{"total":96636764160,"available":96592613376,"internally_reserved":0,"allocated":15130624,"data_stored":14830271,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.38},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.36899999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.40000000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.34399999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.39100000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84899999999999998,"5min":1.462,"15min":1.462},"last":0.65100000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.60399999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.67200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54780,"kb_used_data":15472,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317060,"statfs":{"total":96636764160,"available":96580669440,"internally_reserved":0,"allocated":15843328,"data_stored":15153570,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.36299999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.372,"15min":0.372},"max":{"1min":0.66100000000000003,"5min":0.878,"15min":0.878},"last":0.47799999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.39200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72499999999999998,"5min":1.097,"15min":1.097},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.502}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.63400000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.66200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69764,"kb_used_data":34552,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302076,"statfs":{"total":96636764160,"available":96565325824,"internally_reserved":0,"allocated":35381248,"data_stored":36235028,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.56299999999999994,"5min":0.751,"15min":0.751},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.374},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52100000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.34599999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.505,"15min":0.49099999999999999},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.44400000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.47699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Apr 20 02:54:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52500000000000002,"15min":0.54900000000000004},"min":{"1min":0.40600000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":1.153,"15min":1.153},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.39200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.50900000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52600000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83548,"kb_used_data":48336,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288292,"statfs":{"total":96636764160,"available":96551211008,"internally_reserved":0,"allocated":49496064,"data_stored":51726600,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.628}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.47499999999999998}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.53700000000000003}]}]},{"osd":0,"up_from":15,"seq":64424509502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77752,"kb_used_data":38084,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294088,"statfs":{"total":96636764160,"available":96557146112,"internally_reserved":0,"allocated":38998016,"data_stored":40520952,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.39300000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.314}]},{"osd":4,"last update":"Sat Apr 20 02:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58099999999999996,"15min":0.57999999999999996},"min":{"1min":0.40500000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.57599999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.238,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.52800000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.58099999999999996}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.627}]}]},{"osd":4,"up_from":167,"seq":717259538460,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65960,"kb_used_data":37684,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94305880,"statfs":{"total":96636764160,"available":96569221120,"internally_reserved":0,"allocated":38588416,"data_stored":39829219,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.248,"15min":0.248},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72099999999999997,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.626}]},{"osd":1,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.33300000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53000000000000003,"15min":0.51800000000000002},"min":{"1min":0.45600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72599999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.69399999999999995}]},{"osd":2,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47499999999999998,"15min":0.47099999999999997},"min":{"1min":0.35999999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65500000000000003,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.42099999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.77200000000000002,"5min":1.085,"15min":1.085},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.41999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.83099999999999996,"15min":0.877},"min":{"1min":0.375,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82899999999999996,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.47399999999999998,"15min":0.46600000000000003},"min":{"1min":0.39700000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69099999999999995,"5min":0.753,"15min":0.753},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.376,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.47399999999999998}]},{"osd":6,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.48799999999999999,"15min":0.47599999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.39200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52800000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.38400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.503},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.46500000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:19.020 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-20T02:55:19.348 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:19.349 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":238,"fsid":"a524a365-eaec-4ebd-aa4d-0d76d55a252f","created":"2024-04-20T02:50:07.356902+0000","modified":"2024-04-20T02:55:14.305595+0000","last_up_change":"2024-04-20T02:53:06.173366+0000","last_in_change":"2024-04-20T02:53:55.383050+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-20T02:50:17.498569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-20T02:50:25.955448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-04-20T02:50:33.168857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8dc6e7f3-a5a3-4fd5-91dc-37349c4e8de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":43330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6813","nonce":43330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6815","nonce":43330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":43330}]},"public_addr":"172.21.15.105:6812/43330","cluster_addr":"172.21.15.105:6813/43330","heartbeat_back_addr":"172.21.15.105:6815/43330","heartbeat_front_addr":"172.21.15.105:6814/43330","state":["exists","up"]},{"osd":1,"uuid":"c5ae8a64-fc7c-47ab-af6a-4b762a840645","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":213,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":43331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6805","nonce":43331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6807","nonce":43331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":43331}]},"public_addr":"172.21.15.105:6804/43331","cluster_addr":"172.21.15.105:6805/43331","heartbeat_back_addr":"172.21.15.105:6807/43331","heartbeat_front_addr":"172.21.15.105:6806/43331","state":["exists","up"]},{"osd":2,"uuid":"336e852e-5fa6-4ac0-b053-d31ad40d7c4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":121,"up_from":138,"up_thru":210,"down_at":122,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":66776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6801","nonce":66776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6803","nonce":66776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":66776}]},"public_addr":"172.21.15.105:6800/66776","cluster_addr":"172.21.15.105:6801/66776","heartbeat_back_addr":"172.21.15.105:6803/66776","heartbeat_front_addr":"172.21.15.105:6802/66776","state":["exists","up"]},{"osd":3,"uuid":"695cab3a-74d8-4046-ac61-2656dc0d893e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6809","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6811","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":43329}]},"public_addr":"172.21.15.105:6808/43329","cluster_addr":"172.21.15.105:6809/43329","heartbeat_back_addr":"172.21.15.105:6811/43329","heartbeat_front_addr":"172.21.15.105:6810/43329","state":["exists","up"]},{"osd":4,"uuid":"1d41da44-d894-40ac-acaa-aa0df15296c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":149,"up_from":167,"up_thru":212,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":66272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":66272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6811","nonce":66272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6810","nonce":66272}]},"public_addr":"172.21.15.138:6808/66272","cluster_addr":"172.21.15.138:6809/66272","heartbeat_back_addr":"172.21.15.138:6811/66272","heartbeat_front_addr":"172.21.15.138:6810/66272","state":["exists","up"]},{"osd":5,"uuid":"3e1c33fa-6c8c-4898-99bc-462ecef25f30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":25,"up_from":38,"up_thru":222,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":52590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":52590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6807","nonce":52590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6806","nonce":52590}]},"public_addr":"172.21.15.138:6802/52590","cluster_addr":"172.21.15.138:6804/52590","heartbeat_back_addr":"172.21.15.138:6807/52590","heartbeat_front_addr":"172.21.15.138:6806/52590","state":["exists","up"]},{"osd":6,"uuid":"fd1fc124-7cf8-4628-9076-1f27d6385676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":43329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6801","nonce":43329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":43329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6803","nonce":43329}]},"public_addr":"172.21.15.138:6800/43329","cluster_addr":"172.21.15.138:6801/43329","heartbeat_back_addr":"172.21.15.138:6805/43329","heartbeat_front_addr":"172.21.15.138:6803/43329","state":["exists","up"]},{"osd":7,"uuid":"53ae8193-5c3e-4698-aacd-df2ad1db9143","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":209,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":43351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6813","nonce":43351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6815","nonce":43351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":43351}]},"public_addr":"172.21.15.138:6812/43351","cluster_addr":"172.21.15.138:6813/43351","heartbeat_back_addr":"172.21.15.138:6815/43351","heartbeat_front_addr":"172.21.15.138:6814/43351","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:55:02.698757+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:55:00.447623+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:54:11.526614+0000","dead_epoch":122},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:54:10.018681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:54:58.001669+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:54:56.335507+0000","dead_epoch":26},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:54:55.754614+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-20T02:54:39.905398+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.0","osds":[6,0,2]}],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-20T02:55:20.363 INFO:tasks.ceph:Scrubbing osd.0 2024-04-20T02:55:20.364 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-20T02:55:20.523 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:55:20.523 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-20T02:55:20.523 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:55:20.535 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-20T02:55:20.863 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 0 to deep-scrub 2024-04-20T02:55:20.874 INFO:tasks.ceph:Scrubbing osd.1 2024-04-20T02:55:20.875 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-20T02:55:21.021 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:55:21.021 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-20T02:55:21.021 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:55:21.033 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-20T02:55:21.366 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 1 to deep-scrub 2024-04-20T02:55:21.377 INFO:tasks.ceph:Scrubbing osd.2 2024-04-20T02:55:21.377 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-20T02:55:21.523 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:55:21.524 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-20T02:55:21.524 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:55:21.536 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-20T02:55:21.864 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 2 to deep-scrub 2024-04-20T02:55:21.875 INFO:tasks.ceph:Scrubbing osd.3 2024-04-20T02:55:21.875 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-20T02:55:22.020 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:55:22.021 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-20T02:55:22.021 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:55:22.032 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-20T02:55:22.364 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 3 to deep-scrub 2024-04-20T02:55:22.375 INFO:tasks.ceph:Scrubbing osd.4 2024-04-20T02:55:22.375 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-20T02:55:22.522 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:55:22.522 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-20T02:55:22.522 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:55:22.534 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-20T02:55:22.862 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 4 to deep-scrub 2024-04-20T02:55:22.873 INFO:tasks.ceph:Scrubbing osd.5 2024-04-20T02:55:22.873 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-20T02:55:23.019 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:55:23.020 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-20T02:55:23.020 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:55:23.034 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-20T02:55:23.391 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 5 to deep-scrub 2024-04-20T02:55:23.402 INFO:tasks.ceph:Scrubbing osd.6 2024-04-20T02:55:23.402 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-20T02:55:23.549 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:55:23.549 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-20T02:55:23.549 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:55:23.561 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-20T02:55:23.893 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 6 to deep-scrub 2024-04-20T02:55:23.904 INFO:tasks.ceph:Scrubbing osd.7 2024-04-20T02:55:23.905 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-20T02:55:24.052 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-04-20T02:55:24.052 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-20T02:55:24.052 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-04-20T02:55:24.064 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-20T02:55:24.397 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 7 to deep-scrub 2024-04-20T02:55:24.409 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:24.736 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:24.738 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:24.749 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-04-20T02:55:23.505442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":119920800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":484452,"kb_used_data":204780,"kb_used_omap":15,"kb_used_meta":275952,"kb_avail":754490268,"statfs":{"total":773094113280,"available":772598034432,"internally_reserved":0,"allocated":209694720,"data_stored":215477330,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":282575240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-33,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":18987344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002237"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":212,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.853728+0000","last_change":"2024-04-20T02:54:38.145536+0000","last_active":"2024-04-20T02:55:13.853728+0000","last_peered":"2024-04-20T02:55:13.853728+0000","last_clean":"2024-04-20T02:55:13.853728+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:55:13.853728+0000","last_undegraded":"2024-04-20T02:55:13.853728+0000","last_fullsized":"2024-04-20T02:55:13.853728+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:54:38.145400+0000","last_clean_scrub_stamp":"2024-04-20T02:54:38.145400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:00.806545+0000","scrub_duration":0.0024416220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":282,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:15.420369+0000","last_change":"2024-04-20T02:54:41.862079+0000","last_active":"2024-04-20T02:55:15.420369+0000","last_peered":"2024-04-20T02:55:15.420369+0000","last_clean":"2024-04-20T02:55:15.420369+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:55:15.420369+0000","last_undegraded":"2024-04-20T02:55:15.420369+0000","last_fullsized":"2024-04-20T02:55:15.420369+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:41.861939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:41.861939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0021960899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":292,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:16.087872+0000","last_change":"2024-04-20T02:55:16.087998+0000","last_active":"2024-04-20T02:55:16.087872+0000","last_peered":"2024-04-20T02:55:16.087872+0000","last_clean":"2024-04-20T02:55:16.087872+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:55:16.087872+0000","last_undegraded":"2024-04-20T02:55:16.087872+0000","last_fullsized":"2024-04-20T02:55:16.087872+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:16.087845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:55:16.087845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:43.554780+0000","scrub_duration":0.00229597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"187'261","reported_seq":766,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:14.316425+0000","last_change":"2024-04-20T02:55:14.316425+0000","last_active":"2024-04-20T02:55:14.316425+0000","last_peered":"2024-04-20T02:55:14.316425+0000","last_clean":"2024-04-20T02:55:14.316425+0000","last_became_active":"2024-04-20T02:55:14.316051+0000","last_became_peered":"2024-04-20T02:55:14.316051+0000","last_unstale":"2024-04-20T02:55:14.316425+0000","last_undegraded":"2024-04-20T02:55:14.316425+0000","last_fullsized":"2024-04-20T02:55:14.316425+0000","mapping_epoch":237,"log_start":"180'258","ondisk_log_start":"180'258","created":21,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"187'261","last_scrub_stamp":"2024-04-20T02:55:13.079050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:55:13.079050+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:17.988932+0000","scrub_duration":0.0067537669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":37495155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":48,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.853623+0000","last_change":"2024-04-20T02:55:05.032980+0000","last_active":"2024-04-20T02:55:13.853623+0000","last_peered":"2024-04-20T02:55:13.853623+0000","last_clean":"2024-04-20T02:55:13.853623+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:55:13.853623+0000","last_undegraded":"2024-04-20T02:55:13.853623+0000","last_fullsized":"2024-04-20T02:55:13.853623+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:05.032853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:55:05.032853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:06.092856+0000","scrub_duration":0.0016237370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"192'274","reported_seq":666,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:14.315234+0000","last_change":"2024-04-20T02:54:44.903595+0000","last_active":"2024-04-20T02:55:14.315234+0000","last_peered":"2024-04-20T02:55:14.315234+0000","last_clean":"2024-04-20T02:55:14.315234+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:55:14.315234+0000","last_undegraded":"2024-04-20T02:55:14.315234+0000","last_fullsized":"2024-04-20T02:55:14.315234+0000","mapping_epoch":49,"log_start":"179'271","ondisk_log_start":"179'271","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'274","last_scrub_stamp":"2024-04-20T02:54:44.903413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:44.903413+0000","objects_scrubbed":33,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0062866069999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":417,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:14.189676+0000","last_change":"2024-04-20T02:54:34.852249+0000","last_active":"2024-04-20T02:55:14.189676+0000","last_peered":"2024-04-20T02:55:14.189676+0000","last_clean":"2024-04-20T02:55:14.189676+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:55:14.189676+0000","last_undegraded":"2024-04-20T02:55:14.189676+0000","last_fullsized":"2024-04-20T02:55:14.189676+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:54:34.852096+0000","last_clean_scrub_stamp":"2024-04-20T02:54:34.852096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:42.676440+0000","scrub_duration":0.004799294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":259,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:15.011513+0000","last_change":"2024-04-20T02:54:17.550441+0000","last_active":"2024-04-20T02:55:15.011513+0000","last_peered":"2024-04-20T02:55:15.011513+0000","last_clean":"2024-04-20T02:55:15.011513+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:55:15.011513+0000","last_undegraded":"2024-04-20T02:55:15.011513+0000","last_fullsized":"2024-04-20T02:55:15.011513+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:17.550312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:17.550312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0026229489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"190'241","reported_seq":677,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:15.420399+0000","last_change":"2024-04-20T02:54:05.869200+0000","last_active":"2024-04-20T02:55:15.420399+0000","last_peered":"2024-04-20T02:55:15.420399+0000","last_clean":"2024-04-20T02:55:15.420399+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:55:15.420399+0000","last_undegraded":"2024-04-20T02:55:15.420399+0000","last_fullsized":"2024-04-20T02:55:15.420399+0000","mapping_epoch":213,"log_start":"184'238","ondisk_log_start":"184'238","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'241","last_scrub_stamp":"2024-04-20T02:54:05.869027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:05.869027+0000","objects_scrubbed":30,"log_size":3,"log_dups_size":250,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0062164910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.543204+0000","last_change":"2024-04-20T02:54:12.795718+0000","last_active":"2024-04-20T02:55:13.543204+0000","last_peered":"2024-04-20T02:55:13.543204+0000","last_clean":"2024-04-20T02:55:13.543204+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:13.543204+0000","last_undegraded":"2024-04-20T02:55:13.543204+0000","last_fullsized":"2024-04-20T02:55:13.543204+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:12.795529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:12.795529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:29.282769+0000","scrub_duration":0.037882961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"184'235","reported_seq":689,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-04-20T02:55:13.310744+0000","last_change":"2024-04-20T02:54:46.470707+0000","last_active":"2024-04-20T02:55:13.310744+0000","last_peered":"2024-04-20T02:55:13.310744+0000","last_clean":"2024-04-20T02:55:13.310744+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:13.310744+0000","last_undegraded":"2024-04-20T02:55:13.310744+0000","last_fullsized":"2024-04-20T02:55:13.310744+0000","mapping_epoch":212,"log_start":"182'233","ondisk_log_start":"182'233","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'235","last_scrub_stamp":"2024-04-20T02:54:46.470532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:33.213730+0000","last_clean_scrub_stamp":"2024-04-20T02:54:46.470532+0000","objects_scrubbed":32,"log_size":2,"log_dups_size":239,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:07.306989+0000","scrub_duration":0.0076394719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":226,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:15.011521+0000","last_change":"2024-04-20T02:54:39.951285+0000","last_active":"2024-04-20T02:55:15.011521+0000","last_peered":"2024-04-20T02:55:15.011521+0000","last_clean":"2024-04-20T02:55:15.011521+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:55:15.011521+0000","last_undegraded":"2024-04-20T02:55:15.011521+0000","last_fullsized":"2024-04-20T02:55:15.011521+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:54:39.951168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:50:26.619512+0000","last_clean_scrub_stamp":"2024-04-20T02:54:39.951168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0022092169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":264,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:15.420377+0000","last_change":"2024-04-20T02:55:07.896464+0000","last_active":"2024-04-20T02:55:15.420377+0000","last_peered":"2024-04-20T02:55:15.420377+0000","last_clean":"2024-04-20T02:55:15.420377+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:55:15.420377+0000","last_undegraded":"2024-04-20T02:55:15.420377+0000","last_fullsized":"2024-04-20T02:55:15.420377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:07.896324+0000","last_clean_scrub_stamp":"2024-04-20T02:55:07.896324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.002383745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":119920800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509507,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37508,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334332,"statfs":{"total":96636764160,"available":96598355968,"internally_reserved":0,"allocated":1982464,"data_stored":1390007,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46100000000000002,"15min":0.44900000000000001},"min":{"1min":0.26800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.90300000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.496,"15min":0.46800000000000003},"min":{"1min":0.32200000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.81999999999999995,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.747},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.60899999999999999,"15min":0.623},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85199999999999998,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.625}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.61899999999999999,"15min":0.56399999999999995},"min":{"1min":0.36499999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.0249999999999999,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.3279999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.33900000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47499999999999998,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.48199999999999998}]},{"osd":5,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.39000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.434},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.54200000000000004},"min":{"1min":0.39600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78700000000000003,"5min":2.431,"15min":2.431},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49399999999999999,"15min":0.51500000000000001},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80500000000000005,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.58399999999999996},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94699999999999995,"5min":1.599,"15min":1.599},"last":0.53400000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757305,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52652,"kb_used_data":16376,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94319188,"statfs":{"total":96636764160,"available":96582848512,"internally_reserved":0,"allocated":16769024,"data_stored":16924385,"data_compressed":918439,"data_compressed_allocated":1003520,"data_compressed_original":2293760,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65400000000000003}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.61099999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.84699999999999998,"15min":0.90600000000000003},"min":{"1min":0.32300000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64600000000000002,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.496}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.47499999999999998}]},{"osd":4,"last update":"Sat Apr 20 02:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.47299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35099999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.67600000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.64100000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.58299999999999996}]}]},{"osd":2,"up_from":138,"seq":592705486883,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43116,"kb_used_data":14776,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94328724,"statfs":{"total":96636764160,"available":96592613376,"internally_reserved":0,"allocated":15130624,"data_stored":14830271,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.376,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.38800000000000001,"5min":1.141,"15min":1.141},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39500000000000002,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.495,"5min":0.995,"15min":0.995},"last":0.55500000000000005}]},{"osd":1,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.38100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46200000000000002,"5min":1.012,"15min":1.012},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40699999999999997,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.46899999999999997,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.48799999999999999}]},{"osd":3,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.54600000000000004,"15min":0.58199999999999996},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46400000000000002,"5min":4.609,"15min":4.609},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.66300000000000003,"15min":0.71599999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.42399999999999999}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.39100000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84899999999999998,"5min":1.462,"15min":1.462},"last":0.59299999999999997}]},{"osd":5,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73699999999999999,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.58599999999999997,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":1.115,"15min":1.115},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.62,"15min":0.63400000000000001},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76600000000000001,"5min":1.758,"15min":1.758},"last":0.623},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59299999999999997,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.65900000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.68100000000000005}]},{"osd":7,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.82899999999999996,"15min":0.88300000000000001},"min":{"1min":0.40300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65100000000000002,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79200000000000004,"5min":1.004,"15min":1.004},"last":0.70299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54444,"kb_used_data":14776,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317396,"statfs":{"total":96636764160,"available":96581013504,"internally_reserved":0,"allocated":15130624,"data_stored":14830271,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.29999999999999999,"15min":0.29899999999999999},"min":{"1min":0.183,"5min":0.152,"15min":0.152},"max":{"1min":0.67200000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.34599999999999997},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58899999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.26200000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.29499999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.40799999999999997},"min":{"1min":0.24099999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.504,"5min":1.026,"15min":1.026},"last":0.38},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.30399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69999999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.39900000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.372,"15min":0.372},"max":{"1min":0.66100000000000003,"5min":0.878,"15min":0.878},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.39200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72499999999999998,"5min":1.097,"15min":1.097},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.65100000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.75800000000000001,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.53400000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0349999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.51600000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69792,"kb_used_data":33856,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302048,"statfs":{"total":96636764160,"available":96565297152,"internally_reserved":0,"allocated":34668544,"data_stored":35911729,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.25900000000000001,"5min":0.30399999999999999,"15min":0.30099999999999999},"min":{"1min":0.18099999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.436,"5min":1.323,"15min":1.323},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.57499999999999996,"5min":1.351,"15min":1.351},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.34300000000000003,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.61099999999999999,"5min":0.751,"15min":0.751},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35299999999999998,"15min":0.372},"min":{"1min":0.23699999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.499,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.23999999999999999}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.27500000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.505,"15min":0.49099999999999999},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.46200000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.47699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.44}]},{"osd":6,"last update":"Sat Apr 20 02:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53000000000000003,"15min":0.55100000000000005},"min":{"1min":0.40799999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":1.153,"15min":1.153},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.48999999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.94399999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.38500000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.44800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.748,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67700000000000005,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.41699999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83212,"kb_used_data":47640,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94288628,"statfs":{"total":96636764160,"available":96551555072,"internally_reserved":0,"allocated":48783360,"data_stored":51403301,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.622,"15min":0.79200000000000004},"min":{"1min":0.33100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.01,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48099999999999998,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.747,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.54000000000000004}]},{"osd":1,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73799999999999999,"5min":1.419,"15min":1.419},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.58099999999999996},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.77000000000000002,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.46700000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.39100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.436,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.66900000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.58599999999999997,"15min":0.69899999999999995},"min":{"1min":0.34000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.66800000000000004,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97399999999999998,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.629}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.751}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.65200000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:54:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.51800000000000002,"15min":0.48199999999999998},"min":{"1min":0.32100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.61099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82399999999999995,"5min":3.98,"15min":3.98},"last":0.71399999999999997}]}]},{"osd":0,"up_from":15,"seq":64424509503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77768,"kb_used_data":37736,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294072,"statfs":{"total":96636764160,"available":96557129728,"internally_reserved":0,"allocated":38641664,"data_stored":40358147,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35599999999999998,"15min":0.36099999999999999},"min":{"1min":0.20999999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.149,"5min":1.427,"15min":1.427},"last":0.315},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.2629999999999999,"5min":1.427,"15min":1.427},"last":0.39900000000000002}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.45200000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.215,"15min":0.215},"max":{"1min":1.327,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.556,"5min":1.728,"15min":1.728},"last":0.378}]},{"osd":4,"last update":"Sat Apr 20 02:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58099999999999996,"15min":0.57999999999999996},"min":{"1min":0.40500000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.57599999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.238,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.67700000000000005}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.55600000000000005}]},{"osd":6,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.4730000000000001,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57699999999999996,"15min":0.58499999999999996},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.3029999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.746}]},{"osd":7,"last update":"Sat Apr 20 02:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80900000000000005,"5min":1.268,"15min":1.268},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.622,"15min":0.68700000000000006},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.73799999999999999,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.65900000000000003}]}]},{"osd":4,"up_from":167,"seq":717259538461,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65960,"kb_used_data":37684,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94305880,"statfs":{"total":96636764160,"available":96569221120,"internally_reserved":0,"allocated":38588416,"data_stored":39829219,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.248,"15min":0.248},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.503},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72099999999999997,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.60599999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.33300000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53000000000000003,"15min":0.51800000000000002},"min":{"1min":0.45600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72599999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.64100000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47499999999999998,"15min":0.47099999999999997},"min":{"1min":0.35999999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65500000000000003,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.42099999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.77200000000000002,"5min":1.085,"15min":1.085},"last":0.52900000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.41999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.83099999999999996,"15min":0.877},"min":{"1min":0.375,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82899999999999996,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.66200000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.47399999999999998,"15min":0.46600000000000003},"min":{"1min":0.39700000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69099999999999995,"5min":0.753,"15min":0.753},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.376,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.45000000000000001}]},{"osd":6,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.48799999999999999,"15min":0.47599999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.39200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.38200000000000001}]},{"osd":7,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.38400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.46500000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14839808,"data_stored":16063306,"data_compressed":918439,"data_compressed_allocated":1003520,"data_compressed_original":2293760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:24.754 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-20T02:54:38.145400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=54, tm_sec=38, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.754 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-20T02:54:41.861939+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=54, tm_sec=41, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.754 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-20T02:55:16.087845+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=16, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.754 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-20T02:55:13.079050+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=13, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.755 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-20T02:55:05.032853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=5, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.755 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-20T02:54:44.903413+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=54, tm_sec=44, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.755 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-20T02:54:34.852096+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=54, tm_sec=34, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.755 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-20T02:54:17.550312+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=54, tm_sec=17, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.755 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-20T02:54:05.869027+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=54, tm_sec=5, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.755 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-20T02:54:12.795529+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=54, tm_sec=12, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.755 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-20T02:54:46.470532+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=54, tm_sec=46, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.755 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-20T02:54:39.951168+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=54, tm_sec=39, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.756 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-20T02:55:07.896324+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=7, tm_wday=5, tm_yday=111, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=20, tm_hour=2, tm_min=55, tm_sec=19, tm_wday=5, tm_yday=111, tm_isdst=0) 2024-04-20T02:55:24.756 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-20T02:55:44.756 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-20T02:55:45.096 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-04-20T02:55:45.098 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-04-20T02:55:45.109 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-04-20T02:55:43.509958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98666436,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":1051,"num_read_kb":34779,"num_write":1380,"num_write_kb":326382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":119920800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":484972,"kb_used_data":203388,"kb_used_omap":15,"kb_used_meta":280048,"kb_avail":754489748,"statfs":{"total":773094113280,"available":772597501952,"internally_reserved":0,"allocated":208269312,"data_stored":214816886,"data_compressed":11050677,"data_compressed_allocated":12029952,"data_compressed_original":26972160,"omap_allocated":15992,"internal_metadata":286769544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002251"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":226,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:34.249778+0000","last_change":"2024-04-20T02:55:34.249778+0000","last_active":"2024-04-20T02:55:34.249778+0000","last_peered":"2024-04-20T02:55:34.249778+0000","last_clean":"2024-04-20T02:55:34.249778+0000","last_became_active":"2024-04-20T02:53:56.572279+0000","last_became_peered":"2024-04-20T02:53:56.572279+0000","last_unstale":"2024-04-20T02:55:34.249778+0000","last_undegraded":"2024-04-20T02:55:34.249778+0000","last_fullsized":"2024-04-20T02:55:34.249778+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:34.249626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:34.249626+0000","last_clean_scrub_stamp":"2024-04-20T02:55:34.249626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:49.319110+0000","scrub_duration":0.002581187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":294,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:24.814664+0000","last_change":"2024-04-20T02:55:24.814664+0000","last_active":"2024-04-20T02:55:24.814664+0000","last_peered":"2024-04-20T02:55:24.814664+0000","last_clean":"2024-04-20T02:55:24.814664+0000","last_became_active":"2024-04-20T02:53:55.392801+0000","last_became_peered":"2024-04-20T02:53:55.392801+0000","last_unstale":"2024-04-20T02:55:24.814664+0000","last_undegraded":"2024-04-20T02:55:24.814664+0000","last_fullsized":"2024-04-20T02:55:24.814664+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:24.814557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:24.814557+0000","last_clean_scrub_stamp":"2024-04-20T02:55:24.814557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:41.164457+0000","scrub_duration":0.0019655250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":304,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:33.090968+0000","last_change":"2024-04-20T02:55:33.091100+0000","last_active":"2024-04-20T02:55:33.090968+0000","last_peered":"2024-04-20T02:55:33.090968+0000","last_clean":"2024-04-20T02:55:33.090968+0000","last_became_active":"2024-04-20T02:53:55.396334+0000","last_became_peered":"2024-04-20T02:53:55.396334+0000","last_unstale":"2024-04-20T02:55:33.090968+0000","last_undegraded":"2024-04-20T02:55:33.090968+0000","last_fullsized":"2024-04-20T02:55:33.090968+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:33.090944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:33.090944+0000","last_clean_scrub_stamp":"2024-04-20T02:55:33.090944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:49.656892+0000","scrub_duration":0.002731767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"238'294","reported_seq":813,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:23.085653+0000","last_change":"2024-04-20T02:55:23.085751+0000","last_active":"2024-04-20T02:55:23.085653+0000","last_peered":"2024-04-20T02:55:23.085653+0000","last_clean":"2024-04-20T02:55:23.085653+0000","last_became_active":"2024-04-20T02:55:14.316051+0000","last_became_peered":"2024-04-20T02:55:14.316051+0000","last_unstale":"2024-04-20T02:55:23.085653+0000","last_undegraded":"2024-04-20T02:55:23.085653+0000","last_fullsized":"2024-04-20T02:55:23.085653+0000","mapping_epoch":237,"log_start":"238'285","ondisk_log_start":"238'285","created":21,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'294","last_scrub_stamp":"2024-04-20T02:55:23.085629+0000","last_deep_scrub":"238'294","last_deep_scrub_stamp":"2024-04-20T02:55:23.085629+0000","last_clean_scrub_stamp":"2024-04-20T02:55:23.085629+0000","objects_scrubbed":33,"log_size":9,"log_dups_size":310,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:31.183877+0000","scrub_duration":0.066873620999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28008758,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":245,"num_read_kb":8242,"num_write":315,"num_write_kb":78469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":37495155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:35.268311+0000","last_change":"2024-04-20T02:55:35.268311+0000","last_active":"2024-04-20T02:55:35.268311+0000","last_peered":"2024-04-20T02:55:35.268311+0000","last_clean":"2024-04-20T02:55:35.268311+0000","last_became_active":"2024-04-20T02:53:56.571964+0000","last_became_peered":"2024-04-20T02:53:56.571964+0000","last_unstale":"2024-04-20T02:55:35.268311+0000","last_undegraded":"2024-04-20T02:55:35.268311+0000","last_fullsized":"2024-04-20T02:55:35.268311+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:35.268158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:35.268158+0000","last_clean_scrub_stamp":"2024-04-20T02:55:35.268158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:54.868499+0000","scrub_duration":0.0017992100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"238'307","reported_seq":713,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:23.349521+0000","last_change":"2024-04-20T02:55:23.349521+0000","last_active":"2024-04-20T02:55:23.349521+0000","last_peered":"2024-04-20T02:55:23.349521+0000","last_clean":"2024-04-20T02:55:23.349521+0000","last_became_active":"2024-04-20T02:51:07.118384+0000","last_became_peered":"2024-04-20T02:51:07.118384+0000","last_unstale":"2024-04-20T02:55:23.349521+0000","last_undegraded":"2024-04-20T02:55:23.349521+0000","last_fullsized":"2024-04-20T02:55:23.349521+0000","mapping_epoch":49,"log_start":"238'298","ondisk_log_start":"238'298","created":21,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'307","last_scrub_stamp":"2024-04-20T02:55:23.349315+0000","last_deep_scrub":"238'307","last_deep_scrub_stamp":"2024-04-20T02:55:23.349315+0000","last_clean_scrub_stamp":"2024-04-20T02:55:23.349315+0000","objects_scrubbed":33,"log_size":9,"log_dups_size":306,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:29.516578+0000","scrub_duration":0.068244456999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23752010,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":260,"num_read_kb":13924,"num_write":327,"num_write_kb":94069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":14227195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"228'119","reported_seq":429,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:30.177228+0000","last_change":"2024-04-20T02:55:30.177360+0000","last_active":"2024-04-20T02:55:30.177228+0000","last_peered":"2024-04-20T02:55:30.177228+0000","last_clean":"2024-04-20T02:55:30.177228+0000","last_became_active":"2024-04-20T02:50:19.923034+0000","last_became_peered":"2024-04-20T02:50:19.923034+0000","last_unstale":"2024-04-20T02:55:30.177228+0000","last_undegraded":"2024-04-20T02:55:30.177228+0000","last_fullsized":"2024-04-20T02:55:30.177228+0000","mapping_epoch":15,"log_start":"17'116","ondisk_log_start":"17'116","created":14,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'119","last_scrub_stamp":"2024-04-20T02:55:30.177200+0000","last_deep_scrub":"228'119","last_deep_scrub_stamp":"2024-04-20T02:55:30.177200+0000","last_clean_scrub_stamp":"2024-04-20T02:55:30.177200+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":116,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:38.210838+0000","scrub_duration":0.0043200970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":271,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:23.926330+0000","last_change":"2024-04-20T02:55:23.926330+0000","last_active":"2024-04-20T02:55:23.926330+0000","last_peered":"2024-04-20T02:55:23.926330+0000","last_clean":"2024-04-20T02:55:23.926330+0000","last_became_active":"2024-04-20T02:53:56.572323+0000","last_became_peered":"2024-04-20T02:53:56.572323+0000","last_unstale":"2024-04-20T02:55:23.926330+0000","last_undegraded":"2024-04-20T02:55:23.926330+0000","last_fullsized":"2024-04-20T02:55:23.926330+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:23.926195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:23.926195+0000","last_clean_scrub_stamp":"2024-04-20T02:55:23.926195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:30.958110+0000","scrub_duration":0.0024497249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"238'271","reported_seq":721,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:26.933219+0000","last_change":"2024-04-20T02:55:26.933219+0000","last_active":"2024-04-20T02:55:26.933219+0000","last_peered":"2024-04-20T02:55:26.933219+0000","last_clean":"2024-04-20T02:55:26.933219+0000","last_became_active":"2024-04-20T02:53:59.158075+0000","last_became_peered":"2024-04-20T02:53:59.158075+0000","last_unstale":"2024-04-20T02:55:26.933219+0000","last_undegraded":"2024-04-20T02:55:26.933219+0000","last_fullsized":"2024-04-20T02:55:26.933219+0000","mapping_epoch":213,"log_start":"238'265","ondisk_log_start":"238'265","created":21,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'271","last_scrub_stamp":"2024-04-20T02:55:26.933061+0000","last_deep_scrub":"238'271","last_deep_scrub_stamp":"2024-04-20T02:55:26.933061+0000","last_clean_scrub_stamp":"2024-04-20T02:55:26.933061+0000","objects_scrubbed":30,"log_size":6,"log_dups_size":277,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:44.847958+0000","scrub_duration":0.058666366999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19563304,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":213,"num_read_kb":5887,"num_write":291,"num_write_kb":75589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":29665999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:25.923341+0000","last_change":"2024-04-20T02:55:25.923434+0000","last_active":"2024-04-20T02:55:25.923341+0000","last_peered":"2024-04-20T02:55:25.923341+0000","last_clean":"2024-04-20T02:55:25.923341+0000","last_became_active":"2024-04-20T02:53:56.157056+0000","last_became_peered":"2024-04-20T02:53:56.157056+0000","last_unstale":"2024-04-20T02:55:25.923341+0000","last_undegraded":"2024-04-20T02:55:25.923341+0000","last_fullsized":"2024-04-20T02:55:25.923341+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:25.923316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:25.923316+0000","last_clean_scrub_stamp":"2024-04-20T02:55:25.923316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:31.988300+0000","scrub_duration":0.0014898349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"238'267","reported_seq":735,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:25.811329+0000","last_change":"2024-04-20T02:55:25.811473+0000","last_active":"2024-04-20T02:55:25.811329+0000","last_peered":"2024-04-20T02:55:25.811329+0000","last_clean":"2024-04-20T02:55:25.811329+0000","last_became_active":"2024-04-20T02:53:58.157592+0000","last_became_peered":"2024-04-20T02:53:58.157592+0000","last_unstale":"2024-04-20T02:55:25.811329+0000","last_undegraded":"2024-04-20T02:55:25.811329+0000","last_fullsized":"2024-04-20T02:55:25.811329+0000","mapping_epoch":212,"log_start":"238'259","ondisk_log_start":"238'259","created":21,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'267","last_scrub_stamp":"2024-04-20T02:55:25.811287+0000","last_deep_scrub":"238'267","last_deep_scrub_stamp":"2024-04-20T02:55:25.811287+0000","last_clean_scrub_stamp":"2024-04-20T02:55:25.811287+0000","objects_scrubbed":32,"log_size":8,"log_dups_size":265,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:29.265019+0000","scrub_duration":0.074005722999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26751996,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":217,"num_read_kb":6626,"num_write":288,"num_write_kb":75839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":38532451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":238,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:24.879854+0000","last_change":"2024-04-20T02:55:24.879854+0000","last_active":"2024-04-20T02:55:24.879854+0000","last_peered":"2024-04-20T02:55:24.879854+0000","last_clean":"2024-04-20T02:55:24.879854+0000","last_became_active":"2024-04-20T02:54:08.829281+0000","last_became_peered":"2024-04-20T02:54:08.829281+0000","last_unstale":"2024-04-20T02:55:24.879854+0000","last_undegraded":"2024-04-20T02:55:24.879854+0000","last_fullsized":"2024-04-20T02:55:24.879854+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:24.879711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:24.879711+0000","last_clean_scrub_stamp":"2024-04-20T02:55:24.879711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:35.667220+0000","scrub_duration":0.001487087,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":276,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-04-20T02:55:28.834933+0000","last_change":"2024-04-20T02:55:28.834933+0000","last_active":"2024-04-20T02:55:28.834933+0000","last_peered":"2024-04-20T02:55:28.834933+0000","last_clean":"2024-04-20T02:55:28.834933+0000","last_became_active":"2024-04-20T02:50:28.091208+0000","last_became_peered":"2024-04-20T02:50:28.091208+0000","last_unstale":"2024-04-20T02:55:28.834933+0000","last_undegraded":"2024-04-20T02:55:28.834933+0000","last_fullsized":"2024-04-20T02:55:28.834933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-20T02:55:28.834796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-20T02:55:28.834796+0000","last_clean_scrub_stamp":"2024-04-20T02:55:28.834796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:56:52.625404+0000","scrub_duration":0.001788088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":98076068,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":128,"num_whiteouts":0,"num_read":935,"num_read_kb":34679,"num_write":1221,"num_write_kb":323966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":119920800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194101248,"data_stored":207001914,"data_compressed":11034723,"data_compressed_allocated":11931648,"data_compressed_original":25436160,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":12,"acting":12,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":15954,"data_compressed_allocated":98304,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509511,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37144,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334696,"statfs":{"total":96636764160,"available":96598728704,"internally_reserved":0,"allocated":1982464,"data_stored":1390007,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1589,"internal_metadata":36043211},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.496,"15min":0.46000000000000002},"min":{"1min":0.503,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.77200000000000002,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.52700000000000002,"15min":0.47899999999999998},"min":{"1min":0.439,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.754,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.625}]},{"osd":1,"last update":"Sat Apr 20 02:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.57999999999999996,"15min":0.49399999999999999},"min":{"1min":0.35799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.77000000000000002,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.59099999999999997,"15min":0.61699999999999999},"min":{"1min":0.40200000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69499999999999995,"5min":9.8119999999999994,"15min":9.8119999999999994},"last":0.503}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.45000000000000001,"15min":0.43099999999999999},"min":{"1min":0.314,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.46899999999999997,"15min":0.44500000000000001},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.57399999999999995}]},{"osd":3,"last update":"Sat Apr 20 02:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64000000000000001,"15min":0.57099999999999995},"min":{"1min":0.48999999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.85299999999999998,"5min":9.6440000000000001,"15min":9.6440000000000001},"last":0.495},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.56699999999999995,"15min":0.55100000000000005},"min":{"1min":0.47499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73099999999999998,"5min":3.8599999999999999,"15min":3.8599999999999999},"last":0.59299999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.33900000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.47499999999999998,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.37}]},{"osd":5,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.39000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.441},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.51600000000000001,"15min":0.54200000000000004},"min":{"1min":0.39600000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.78700000000000003,"5min":2.431,"15min":2.431},"last":0.52700000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.502,"15min":0.51800000000000002},"min":{"1min":0.42099999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.84899999999999998,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54400000000000004,"15min":0.58199999999999996},"min":{"1min":0.41399999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80100000000000005,"5min":1.599,"15min":1.599},"last":0.46999999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757309,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52304,"kb_used_data":16028,"kb_used_omap":1,"kb_used_meta":35902,"kb_avail":94319536,"statfs":{"total":96636764160,"available":96583204864,"internally_reserved":0,"allocated":16412672,"data_stored":16759274,"data_compressed":918439,"data_compressed_allocated":1003520,"data_compressed_original":2293760,"omap_allocated":1657,"internal_metadata":36764039},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.44900000000000001,"15min":0.40500000000000003},"min":{"1min":0.33300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.47399999999999998,"15min":0.434},"min":{"1min":0.49099999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76200000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.69599999999999995}]},{"osd":1,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.50600000000000001,"15min":0.45200000000000001},"min":{"1min":0.35099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69999999999999996,"5min":4.1719999999999997,"15min":4.1719999999999997},"last":0.746},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.504,"15min":0.44400000000000001},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.80400000000000005,"5min":4.2030000000000003,"15min":4.2030000000000003},"last":0.73299999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.77600000000000002,"15min":0.88200000000000001},"min":{"1min":0.42099999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.86899999999999999,"5min":9.7829999999999995,"15min":9.7829999999999995},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.52900000000000003,"15min":0.51300000000000001},"min":{"1min":0.35399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.60699999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.47499999999999998},"min":{"1min":0.432,"5min":0.31,"15min":0.31},"max":{"1min":0.67600000000000005,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.38100000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.71999999999999997}]},{"osd":4,"last update":"Sat Apr 20 02:55:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.45300000000000001,"15min":0.44400000000000001},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.47299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35099999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.622}]},{"osd":6,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.41799999999999998},"min":{"1min":0.39800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.66800000000000004,"5min":1.472,"15min":1.472},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.46999999999999997,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.75800000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.54000000000000004}]},{"osd":7,"last update":"Sat Apr 20 02:55:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.48499999999999999,"15min":0.45100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.495},"min":{"1min":0.46899999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.66000000000000003}]}]},{"osd":2,"up_from":138,"seq":592705486887,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42752,"kb_used_data":14776,"kb_used_omap":2,"kb_used_meta":27965,"kb_avail":94329088,"statfs":{"total":96636764160,"available":96592986112,"internally_reserved":0,"allocated":15130624,"data_stored":14830271,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":2595,"internal_metadata":28636637},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.36699999999999999,"15min":0.38600000000000001},"min":{"1min":0.23799999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.51200000000000001,"5min":1.141,"15min":1.141},"last":0.373},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.39800000000000002,"15min":0.40100000000000002},"min":{"1min":0.30499999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.55500000000000005,"5min":0.995,"15min":0.995},"last":0.42299999999999999}]},{"osd":1,"last update":"Sat Apr 20 02:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.254,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.46500000000000002,"5min":1.012,"15min":1.012},"last":0.318},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.41299999999999998,"15min":0.41699999999999998},"min":{"1min":0.247,"5min":0.218,"15min":0.218},"max":{"1min":0.59699999999999998,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39900000000000002}]},{"osd":3,"last update":"Sat Apr 20 02:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.501,"15min":0.56699999999999995},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56999999999999995,"5min":4.609,"15min":4.609},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.58799999999999997,"15min":0.69099999999999995},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.46100000000000002,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.38}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68200000000000005,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.622},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.39100000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84899999999999998,"5min":1.462,"15min":1.462},"last":0.497}]},{"osd":5,"last update":"Sat Apr 20 02:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55500000000000005,"15min":0.56000000000000005},"min":{"1min":0.442,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82999999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.76800000000000002,"5min":1.115,"15min":1.115},"last":0.55000000000000004}]},{"osd":6,"last update":"Sat Apr 20 02:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.61099999999999999,"15min":0.63100000000000001},"min":{"1min":0.42099999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78300000000000003,"5min":1.758,"15min":1.758},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58999999999999997,"15min":0.60599999999999998},"min":{"1min":0.503,"5min":0.372,"15min":0.372},"max":{"1min":0.86399999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Apr 20 02:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.75900000000000001,"15min":0.85999999999999999},"min":{"1min":0.42499999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.91500000000000004,"5min":9.1419999999999995,"15min":9.1419999999999995},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.47499999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.874,"5min":1.004,"15min":1.004},"last":0.57899999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54096,"kb_used_data":14428,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317744,"statfs":{"total":96636764160,"available":96581369856,"internally_reserved":0,"allocated":14774272,"data_stored":14665160,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.30499999999999999,"15min":0.30099999999999999},"min":{"1min":0.23300000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.49099999999999999,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.34399999999999997},"min":{"1min":0.22,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57799999999999996,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.437}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.307,"15min":0.30499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32100000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48099999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Apr 20 02:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.36299999999999999,"15min":0.40300000000000002},"min":{"1min":0.23599999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.50700000000000001,"5min":1.026,"15min":1.026},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39800000000000002,"15min":0.40000000000000002},"min":{"1min":0.30499999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.59199999999999997,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.45800000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:55:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.372,"15min":0.372},"max":{"1min":0.66100000000000003,"5min":0.878,"15min":0.878},"last":0.65200000000000002}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53000000000000003,"15min":0.55000000000000004},"min":{"1min":0.39200000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72499999999999998,"5min":1.097,"15min":1.097},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79600000000000004,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.70699999999999996}]},{"osd":6,"last update":"Sat Apr 20 02:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.53900000000000003,"15min":0.53400000000000003},"min":{"1min":0.47699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.62},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.36599999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73099999999999998,"5min":3.2719999999999998,"15min":3.2719999999999998},"last":0.73899999999999999}]},{"osd":7,"last update":"Sat Apr 20 02:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55300000000000005,"15min":0.54400000000000004},"min":{"1min":0.45500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84499999999999997,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56399999999999995,"15min":0.58599999999999997},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0149999999999999,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.69399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69080,"kb_used_data":33508,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94302760,"statfs":{"total":96636764160,"available":96566026240,"internally_reserved":0,"allocated":34312192,"data_stored":35746618,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":1592,"internal_metadata":36043208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.309,"15min":0.30299999999999999},"min":{"1min":0.22600000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.44900000000000001,"5min":1.323,"15min":1.323},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39100000000000001,"15min":0.38500000000000001},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.53200000000000003,"5min":1.351,"15min":1.351},"last":0.41699999999999998}]},{"osd":1,"last update":"Sat Apr 20 02:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.34300000000000003,"15min":0.34100000000000003},"min":{"1min":0.28799999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.61099999999999999,"5min":0.751,"15min":0.751},"last":0.439},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35299999999999998,"15min":0.372},"min":{"1min":0.23699999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.499,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Apr 20 02:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.53600000000000003,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.438,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.45700000000000002,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.505,"15min":0.49099999999999999},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.44400000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.69399999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.49199999999999999},"min":{"1min":0.41199999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.5,"15min":0.48399999999999999},"min":{"1min":0.47699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.621}]},{"osd":6,"last update":"Sat Apr 20 02:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53000000000000003,"15min":0.55100000000000005},"min":{"1min":0.40799999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77000000000000002,"5min":1.153,"15min":1.153},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.48999999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.94399999999999995,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.53400000000000003}]},{"osd":7,"last update":"Sat Apr 20 02:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.97899999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.41699999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.56100000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86960,"kb_used_data":47292,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94284880,"statfs":{"total":96636764160,"available":96547717120,"internally_reserved":0,"allocated":48427008,"data_stored":51238190,"data_compressed":2657395,"data_compressed_allocated":2908160,"data_compressed_original":6762496,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.77300000000000002},"min":{"1min":0.29599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.84199999999999997,"5min":9.9900000000000002,"15min":9.9900000000000002},"last":0.62},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.51600000000000001,"15min":0.45800000000000002},"min":{"1min":0.373,"5min":0.223,"15min":0.223},"max":{"1min":0.80700000000000005,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.69999999999999996}]},{"osd":1,"last update":"Sat Apr 20 02:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.503,"15min":0.48899999999999999},"min":{"1min":0.39200000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77300000000000002,"5min":1.419,"15min":1.419},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52300000000000002,"15min":0.57699999999999996},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.749,"5min":4.0369999999999999,"15min":4.0369999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Apr 20 02:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.47599999999999998},"min":{"1min":0.41299999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.56200000000000006,"15min":0.53800000000000003},"min":{"1min":0.48099999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.72999999999999998}]},{"osd":3,"last update":"Sat Apr 20 02:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54300000000000004,"15min":0.68500000000000005},"min":{"1min":0.40500000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.749,"5min":4.0099999999999998,"15min":4.0099999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61499999999999999,"15min":0.79300000000000004},"min":{"1min":0.33900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.878,"5min":9.9570000000000007,"15min":9.9570000000000007},"last":0.68700000000000006}]},{"osd":4,"last update":"Sat Apr 20 02:55:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.307,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.36199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83299999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Apr 20 02:55:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45200000000000001},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.77300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.35999999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.83699999999999997,"5min":1.038,"15min":1.038},"last":0.56699999999999995}]},{"osd":7,"last update":"Sat Apr 20 02:55:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.54500000000000004,"15min":0.49099999999999999},"min":{"1min":0.45100000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.78100000000000003,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55400000000000005,"15min":0.60899999999999999},"min":{"1min":0.34599999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80800000000000005,"5min":3.98,"15min":3.98},"last":0.60899999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509507,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77040,"kb_used_data":37736,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94294800,"statfs":{"total":96636764160,"available":96557875200,"internally_reserved":0,"allocated":38641664,"data_stored":40358147,"data_compressed":2051833,"data_compressed_allocated":2240512,"data_compressed_original":5124096,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Apr 20 02:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.35799999999999998,"15min":0.36199999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.65900000000000003,"5min":1.427,"15min":1.427},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.37,"15min":0.38},"min":{"1min":0.223,"5min":0.183,"15min":0.183},"max":{"1min":0.63200000000000001,"5min":1.427,"15min":1.427},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Apr 20 02:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38,"15min":0.375},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1899999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Apr 20 02:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.42499999999999999,"15min":0.41099999999999998},"min":{"1min":0.248,"5min":0.215,"15min":0.215},"max":{"1min":0.57099999999999995,"5min":3.2730000000000001,"15min":3.2730000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.42399999999999999,"15min":0.42699999999999999},"min":{"1min":0.29399999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59699999999999998,"5min":1.728,"15min":1.728},"last":0.51800000000000002}]},{"osd":4,"last update":"Sat Apr 20 02:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58099999999999996,"15min":0.57999999999999996},"min":{"1min":0.40500000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58099999999999996,"15min":0.57599999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.238,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Apr 20 02:54:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.47099999999999997,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.179,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.46300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.69399999999999995}]},{"osd":6,"last update":"Sat Apr 20 02:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55900000000000005,"15min":0.58499999999999996},"min":{"1min":0.36799999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70699999999999996,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58099999999999996,"15min":0.58699999999999997},"min":{"1min":0.43099999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85699999999999998,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.64400000000000002}]},{"osd":7,"last update":"Sat Apr 20 02:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.57599999999999996,"15min":0.56499999999999995},"min":{"1min":0.436,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77500000000000002,"5min":1.268,"15min":1.268},"last":0.629},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60699999999999998,"15min":0.68200000000000005},"min":{"1min":0.42399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.91200000000000003,"5min":4.1449999999999996,"15min":4.1449999999999996},"last":0.61399999999999999}]}]},{"osd":4,"up_from":167,"seq":717259538465,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65596,"kb_used_data":37684,"kb_used_omap":3,"kb_used_meta":27900,"kb_avail":94306244,"statfs":{"total":96636764160,"available":96569593856,"internally_reserved":0,"allocated":38588416,"data_stored":39829219,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":3778,"internal_metadata":28569918},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.28100000000000003,"5min":0.248,"15min":0.248},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72099999999999997,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.45000000000000001}]},{"osd":1,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.33300000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53000000000000003,"15min":0.51800000000000002},"min":{"1min":0.45600000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.72599999999999998,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.47499999999999998,"15min":0.47099999999999997},"min":{"1min":0.35999999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.65500000000000003,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.42099999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.77200000000000002,"5min":1.085,"15min":1.085},"last":0.54400000000000004}]},{"osd":3,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.41999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.83099999999999996,"15min":0.877},"min":{"1min":0.375,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82899999999999996,"5min":6.1210000000000004,"15min":6.1210000000000004},"last":0.53100000000000003}]},{"osd":5,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.47399999999999998,"15min":0.46600000000000003},"min":{"1min":0.39700000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69099999999999995,"5min":0.753,"15min":0.753},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.45900000000000002,"15min":0.44500000000000001},"min":{"1min":0.376,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.40200000000000002}]},{"osd":6,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.48799999999999999,"15min":0.47599999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.39200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.495}]},{"osd":7,"last update":"Sat Apr 20 02:55:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.38400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.53100000000000003,"15min":0.52100000000000002},"min":{"1min":0.46500000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5318,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13969192,"data_compressed":713287,"data_compressed_allocated":765952,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":35050650,"data_compressed":1944603,"data_compressed_allocated":2105344,"data_compressed_original":4521984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36659200,"data_stored":38968140,"data_compressed":2046515,"data_compressed_allocated":2207744,"data_compressed_original":4612096,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14839808,"data_stored":16063306,"data_compressed":918439,"data_compressed_allocated":1003520,"data_compressed_original":2293760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46800896,"data_stored":50013294,"data_compressed":2652077,"data_compressed_allocated":2875392,"data_compressed_original":6250496,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-20T02:55:45.112 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-20T02:55:45.459 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-20T02:55:45.460 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-20T02:55:45.460 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-20T02:55:45.460 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.507 INFO:tasks.ceph.osd.0:Stopped 2024-04-20T02:55:45.507 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-20T02:55:45.507 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.543 INFO:tasks.ceph.osd.1:Stopped 2024-04-20T02:55:45.543 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-20T02:55:45.543 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.607 INFO:tasks.ceph.osd.2:Stopped 2024-04-20T02:55:45.607 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-20T02:55:45.607 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.675 INFO:tasks.ceph.osd.3:Stopped 2024-04-20T02:55:45.675 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-20T02:55:45.675 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.722 INFO:tasks.ceph.osd.4:Stopped 2024-04-20T02:55:45.723 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-20T02:55:45.723 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.754 INFO:tasks.ceph.osd.5:Stopped 2024-04-20T02:55:45.754 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-20T02:55:45.754 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.823 INFO:tasks.ceph.osd.6:Stopped 2024-04-20T02:55:45.823 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-20T02:55:45.823 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.897 INFO:tasks.ceph.osd.7:Stopped 2024-04-20T02:55:45.897 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-20T02:55:45.897 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-20T02:55:45.898 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.920 INFO:tasks.ceph.mgr.y:Stopped 2024-04-20T02:55:45.920 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-20T02:55:45.920 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.965 INFO:tasks.ceph.mgr.x:Stopped 2024-04-20T02:55:45.965 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-20T02:55:45.966 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-20T02:55:45.966 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:45.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-04-20T02:55:45.977 INFO:tasks.ceph.mon.a:Stopped 2024-04-20T02:55:45.977 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-20T02:55:45.977 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:46.028 INFO:tasks.ceph.mon.c:Stopped 2024-04-20T02:55:46.028 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-20T02:55:46.029 INFO:teuthology.orchestra.run:waiting for 300 2024-04-20T02:55:46.041 INFO:tasks.ceph.mon.b:Stopped 2024-04-20T02:55:46.041 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-20T02:55:46.041 DEBUG:teuthology.orchestra.run.smithi105:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-20T02:55:46.100 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:55:46.100 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-20T02:55:46.356 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:55:46.357 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-20T02:55:46.484 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:55:46.484 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-20T02:55:46.618 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:55:46.619 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-20T02:55:46.704 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:55:46.704 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-20T02:55:46.861 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:55:46.861 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-20T02:55:46.934 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:55:46.935 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-20T02:55:47.044 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:55:47.044 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-20T02:55:47.121 INFO:tasks.ceph:Archiving mon data... 2024-04-20T02:55:47.122 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272/data/mon.a.tgz 2024-04-20T02:55:47.122 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-04-20T02:55:47.140 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.mKb6orNjq9 2024-04-20T02:55:47.140 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.mKb6orNjq9 2024-04-20T02:55:47.322 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.mKb6orNjq9 2024-04-20T02:55:47.409 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.mKb6orNjq9 is 548KB 2024-04-20T02:55:47.512 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.mKb6orNjq9 2024-04-20T02:55:47.529 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272/data/mon.c.tgz 2024-04-20T02:55:47.529 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-04-20T02:55:47.585 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.JSiwujFDq1 2024-04-20T02:55:47.585 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.JSiwujFDq1 2024-04-20T02:55:47.744 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.JSiwujFDq1 2024-04-20T02:55:47.830 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.JSiwujFDq1 is 548KB 2024-04-20T02:55:47.931 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.JSiwujFDq1 2024-04-20T02:55:47.947 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272/data/mon.b.tgz 2024-04-20T02:55:47.947 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-04-20T02:55:47.968 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.A4F8kZZNji 2024-04-20T02:55:47.968 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.A4F8kZZNji 2024-04-20T02:55:48.157 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.A4F8kZZNji 2024-04-20T02:55:48.243 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.A4F8kZZNji is 548KB 2024-04-20T02:55:48.342 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.A4F8kZZNji 2024-04-20T02:55:48.358 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-20T02:55:48.359 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-20T02:55:48.361 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-20T02:55:48.424 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-20T02:55:48.424 DEBUG:teuthology.orchestra.run.smithi105:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-20T02:55:48.428 DEBUG:teuthology.orchestra.run.smithi138:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-20T02:55:48.453 INFO:teuthology.orchestra.run.smithi105.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-20T02:55:48.491 INFO:teuthology.orchestra.run.smithi138.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-20T02:55:48.497 INFO:tasks.ceph:Archiving crash dumps... 2024-04-20T02:55:48.499 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272/remote/smithi105/crash 2024-04-20T02:55:48.500 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-20T02:55:48.528 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272/remote/smithi138/crash 2024-04-20T02:55:48.529 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-20T02:55:48.563 INFO:tasks.ceph:Compressing logs... 2024-04-20T02:55:48.564 DEBUG:teuthology.orchestra.run.smithi105:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T02:55:48.572 DEBUG:teuthology.orchestra.run.smithi138:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T02:55:50.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-04-20T02:55:57.892 INFO:tasks.ceph:Archiving logs... 2024-04-20T02:55:57.892 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272/remote/smithi105/log 2024-04-20T02:55:57.893 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T02:55:59.463 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272/remote/smithi138/log 2024-04-20T02:55:59.464 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-20T02:56:00.694 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-20T02:56:00.704 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-20T02:56:00.705 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T02:56:00.707 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-20T02:56:00.791 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-20T02:56:00.802 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-20T02:56:00.802 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T02:56:00.805 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-20T02:56:00.821 INFO:teuthology.orchestra.run.smithi105.stderr:bash: ntpq: command not found 2024-04-20T02:56:00.823 INFO:teuthology.orchestra.run.smithi138.stderr:bash: ntpq: command not found 2024-04-20T02:56:00.827 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T02:56:00.827 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-04-20T02:56:00.827 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 -1362us[-1362us] +/- 92ms 2024-04-20T02:56:00.827 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -116us[ -153us] +/- 65ms 2024-04-20T02:56:00.827 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +1323us[+1323us] +/- 80ms 2024-04-20T02:56:00.827 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:56:00.828 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-20T02:56:00.828 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-04-20T02:56:00.828 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 -1341us[-1343us] +/- 92ms 2024-04-20T02:56:00.828 INFO:teuthology.orchestra.run.smithi138.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -89us[ -91us] +/- 65ms 2024-04-20T02:56:00.829 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +1361us[+1359us] +/- 80ms 2024-04-20T02:56:00.829 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-20T02:56:00.829 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-20T02:56:00.839 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-20T02:56:00.840 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-20T02:56:00.849 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T02:56:00.958 DEBUG:teuthology.orchestra.run.smithi138:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-20T02:56:01.042 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-20T02:56:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:56:01.075 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-20T02:56:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:56:01.108 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-20T02:56:01.118 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-20T02:56:01.134 INFO:teuthology.task.internal:Duration was 1463.704077 seconds 2024-04-20T02:56:01.135 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-20T02:56:01.144 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-20T02:56:01.144 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T02:56:01.146 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-20T02:56:01.194 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T02:56:01.196 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-20T02:56:01.420 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-20T02:56:01.420 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-04-20T02:56:01.420 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T02:56:01.443 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2024-04-20T02:56:01.443 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-20T02:56:01.464 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-20T02:56:01.464 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T02:56:01.485 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-20T02:56:01.551 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-20T02:56:01.552 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T02:56:01.554 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-20T02:56:01.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-20T02:56:01.660 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-20T02:56:01.660 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T02:56:01.687 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-20T02:56:01.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-20T02:56:01.724 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T02:56:01.729 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-20T02:56:01.753 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-04-20T02:56:01.780 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2024-04-20T02:56:01.824 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T02:56:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:56:01.842 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-20T02:56:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-20T02:56:01.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-20T02:56:01.893 INFO:teuthology.task.internal:Transferring archived files... 2024-04-20T02:56:01.894 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272/remote/smithi105 2024-04-20T02:56:01.894 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T02:56:01.935 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-20_01:10:46-rados-wip-yuri7-testing-2024-04-18-1351-reef-distro-default-smithi/7664272/remote/smithi138 2024-04-20T02:56:01.936 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-20T02:56:01.977 INFO:teuthology.task.internal:Removing archive directory... 2024-04-20T02:56:01.977 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T02:56:01.980 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-20T02:56:02.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-20T02:56:02.036 INFO:teuthology.task.internal:Not uploading archives. 2024-04-20T02:56:02.036 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-20T02:56:02.047 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-20T02:56:02.047 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T02:56:02.049 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-20T02:56:02.063 INFO:teuthology.orchestra.run.smithi105.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 20 02:56 /home/ubuntu/cephtest 2024-04-20T02:56:02.082 INFO:teuthology.orchestra.run.smithi138.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 20 02:56 /home/ubuntu/cephtest 2024-04-20T02:56:02.083 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-20T02:56:02.094 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-20T02:56:02.120 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-snaps} duration: 1463.7040767669678 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-20T02:56:02.121 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-20T02:56:02.198 INFO:teuthology.run:pass